Feature Selection Techniques Cheat Sheet

Summarizes filter, wrapper, and embedded methods for selecting the most predictive features, with scikit-learn code for each approach.

2 PagesIntermediateMar 8, 2026

Filter Methods

Score features independently of any model before training.

Variance threshold- Removes low-variance (near-constant) features that carry little information
Correlation coefficient- Drops features highly correlated with each other to reduce redundancy
Chi-squared test- Measures dependence between categorical features and a categorical target
ANOVA F-test- Scores how well each numeric feature separates the classes of a categorical target
Mutual information- Captures both linear and non-linear dependence between a feature and the target

Filter Selection with SelectKBest

Keep the k highest-scoring features using a statistical test.

python

from sklearn.feature_selection import SelectKBest, f_classifselector = SelectKBest(score_func=f_classif, k=10)X_new = selector.fit_transform(X_train, y_train)# Get the names of the selected columnsselected_cols = X_train.columns[selector.get_support()]print(selected_cols.tolist())

Wrapper Method: RFE

Recursive Feature Elimination trains a model repeatedly, dropping the weakest feature each round.

python

from sklearn.feature_selection import RFEfrom sklearn.linear_model import LogisticRegressionestimator = LogisticRegression(max_iter=1000)rfe = RFE(estimator, n_features_to_select=8, step=1)rfe.fit(X_train, y_train)print(X_train.columns[rfe.support_])   # Selected featuresprint(rfe.ranking_)                    # 1 = selected, higher = eliminated later

Embedded Method: L1 Regularization

Lasso drives irrelevant feature coefficients to exactly zero during training.

python

from sklearn.linear_model import LassoCVimport numpy as nplasso = LassoCV(cv=5, random_state=42).fit(X_train, y_train)importance = np.abs(lasso.coef_)selected = X_train.columns[importance > 0]print(selected.tolist())

Choosing a Method

Trade-offs between the three families.

Filter- Fastest, model-agnostic, good for a first pass on high-dimensional data
Wrapper- Most accurate for a specific model but computationally expensive (trains many models)
Embedded- Balances speed and accuracy by folding selection into model training (Lasso, tree feature_importances_)
Multicollinearity check- Use Variance Inflation Factor (VIF > 10 is often flagged) before filter methods relying on correlation

Pro Tip

Always fit feature selectors only on the training fold inside cross-validation -- selecting features on the full dataset first leaks target information and inflates your reported accuracy.

Was this cheat sheet helpful?

Data Warehousing Concepts Cheat Sheet

Outlier Detection Cheat Sheet

Explore Topics

#FeatureSelectionTechniques#FeatureSelectionTechniquesCheatSheet#DataScience#Intermediate#FilterMethods#FilterSelectionWithSelectKBest#WrapperMethodRFE#EmbeddedMethodL1Regularization#Functions#MachineLearning#CheatSheet#SkillVeris

Sri Hayavadhana Info-Tech

Professional Web Designing Services