Feature Selection Techniques Cheat Sheet
Summarizes filter, wrapper, and embedded methods for selecting the most predictive features, with scikit-learn code for each approach.
2 PagesIntermediateMar 8, 2026
Filter Methods
Score features independently of any model before training.
- Variance threshold- Removes low-variance (near-constant) features that carry little information
- Correlation coefficient- Drops features highly correlated with each other to reduce redundancy
- Chi-squared test- Measures dependence between categorical features and a categorical target
- ANOVA F-test- Scores how well each numeric feature separates the classes of a categorical target
- Mutual information- Captures both linear and non-linear dependence between a feature and the target
Filter Selection with SelectKBest
Keep the k highest-scoring features using a statistical test.
python
from sklearn.feature_selection import SelectKBest, f_classifselector = SelectKBest(score_func=f_classif, k=10)X_new = selector.fit_transform(X_train, y_train)# Get the names of the selected columnsselected_cols = X_train.columns[selector.get_support()]print(selected_cols.tolist())
Wrapper Method: RFE
Recursive Feature Elimination trains a model repeatedly, dropping the weakest feature each round.
python
from sklearn.feature_selection import RFEfrom sklearn.linear_model import LogisticRegressionestimator = LogisticRegression(max_iter=1000)rfe = RFE(estimator, n_features_to_select=8, step=1)rfe.fit(X_train, y_train)print(X_train.columns[rfe.support_]) # Selected featuresprint(rfe.ranking_) # 1 = selected, higher = eliminated later
Embedded Method: L1 Regularization
Lasso drives irrelevant feature coefficients to exactly zero during training.
python
from sklearn.linear_model import LassoCVimport numpy as nplasso = LassoCV(cv=5, random_state=42).fit(X_train, y_train)importance = np.abs(lasso.coef_)selected = X_train.columns[importance > 0]print(selected.tolist())
Choosing a Method
Trade-offs between the three families.
- Filter- Fastest, model-agnostic, good for a first pass on high-dimensional data
- Wrapper- Most accurate for a specific model but computationally expensive (trains many models)
- Embedded- Balances speed and accuracy by folding selection into model training (Lasso, tree feature_importances_)
- Multicollinearity check- Use Variance Inflation Factor (VIF > 10 is often flagged) before filter methods relying on correlation
Mutual Information + SelectPercentile
Rank features by non-linear dependence on the target and keep the top percentile instead of a fixed k.
python
from sklearn.feature_selection import mutual_info_classif, SelectPercentileselector = SelectPercentile( score_func=lambda X, y: mutual_info_classif(X, y, random_state=42), percentile=25,)X_new = selector.fit_transform(X_train, y_train)scores = selector.scores_ranked = sorted(zip(X_train.columns, scores), key=lambda t: -t[1])print(ranked[:10])
Permutation Importance (Model-Agnostic)
Measure how much a fitted model's score drops when a feature's values are shuffled, avoiding the bias of impurity-based importances toward high-cardinality features.
python
from sklearn.inspection import permutation_importancefrom sklearn.ensemble import RandomForestClassifiermodel = RandomForestClassifier(n_estimators=300, random_state=42).fit(X_train, y_train)result = permutation_importance( model, X_valid, y_valid, n_repeats=20, random_state=42, scoring='roc_auc')importances = result.importances_mean - 2 * result.importances_stdkeep = X_train.columns[importances > 0] # drop features indistinguishable from noiseprint(keep.tolist())
RFECV: Recursive Elimination with Cross-Validation
Let cross-validated scoring pick the optimal number of features instead of guessing n_features_to_select.
python
from sklearn.feature_selection import RFECVfrom sklearn.ensemble import GradientBoostingClassifierfrom sklearn.model_selection import StratifiedKFoldrfecv = RFECV( estimator=GradientBoostingClassifier(random_state=42), step=1, cv=StratifiedKFold(5), scoring='f1', min_features_to_select=5, n_jobs=-1,)rfecv.fit(X_train, y_train)print(f"Optimal feature count: {rfecv.n_features_}")print(X_train.columns[rfecv.support_].tolist())
Sequential Feature Selector (Forward/Backward)
Greedily add or remove features by cross-validated score when RFE's coefficient/importance ranking isn't available for the estimator.
python
from sklearn.feature_selection import SequentialFeatureSelectorfrom sklearn.neighbors import KNeighborsClassifiersfs = SequentialFeatureSelector( KNeighborsClassifier(n_neighbors=15), n_features_to_select='auto', tol=0.001, direction='forward', # or 'backward' cv=5, n_jobs=-1,)sfs.fit(X_train, y_train)print(X_train.columns[sfs.get_support()].tolist())
Advanced Selection Strategies
Techniques beyond the standard filter/wrapper/embedded trio for high-dimensional or correlated feature spaces.
- Boruta- Wraps a random forest against shadow (shuffled) copies of every feature and keeps only those that consistently beat their shadow's importance
- mRMR (min-redundancy max-relevance)- Greedily selects features that are individually predictive but mutually redundant with already-chosen features, penalized
- Stability selection- Runs Lasso/RFE across many bootstrap resamples and keeps features selected in a high fraction of runs, guarding against selection instability
- SHAP-based pruning- Fits a full model, ranks features by mean absolute SHAP value, and drops the long tail that contributes near-zero marginal signal
- Variance Inflation Factor (VIF) pruning- Iteratively drops the feature with the highest VIF (regressing it on all others) until every remaining VIF is below threshold
- Hierarchical clustering on |correlation|- Clusters correlated features via 1 - |corr| distance and keeps one representative per cluster instead of a pairwise threshold
Pro Tip
Always fit feature selectors only on the training fold inside cross-validation -- selecting features on the full dataset first leaks target information and inflates your reported accuracy.
Was this cheat sheet helpful?
Explore Topics
#FeatureSelectionTechniques#FeatureSelectionTechniquesCheatSheet#DataScience#Intermediate#FilterMethods#FilterSelectionWithSelectKBest#WrapperMethodRFE#EmbeddedMethodL1Regularization#Functions#MachineLearning#CheatSheet#SkillVeris
