Feature Selection Techniques
Feature selection is the process of identifying and keeping only the subset of available features that contribute meaningfully to a model's predictive performance, while discarding redundant, irrelevant, or noisy ones. It matters for several reasons: fewer irrelevant features reduce the risk of overfitting (the model has less opportunity to fit noise), models train and predict faster, and a smaller, well-chosen feature set is easier for humans to interpret and explain — an important property in regulated domains like finance and healthcare.
Cricket analogy: A selector picks Virat Kohli, Jasprit Bumrah, and Ravindra Jadeja for a must-win match instead of fielding all 30 contracted players, since extra untested names add risk without adding runs or wickets.
Filter Methods
Filter methods score each feature independently of any specific model, typically using a statistical measure of association with the target — correlation coefficients for numeric targets, chi-squared or mutual information for categorical targets — and then keep the top-scoring features. They are fast because they never train a model, which makes them a good first-pass, model-agnostic screen, but their weakness is that they evaluate features one at a time and can miss features that are only useful in combination with others (interaction effects) or wrongly keep features that are individually correlated with the target but redundant with each other.
Cricket analogy: A scout ranks batsmen purely by career batting average without watching how they play together in a partnership, so it's a quick first cut but misses two openers who click brilliantly as a pair.
Wrapper and Embedded Methods
Wrapper methods evaluate subsets of features by actually training and validating a model on each candidate subset, searching for the combination that performs best. Recursive Feature Elimination (RFE) is a common wrapper approach: it trains a model, ranks features by importance, drops the weakest one, and repeats until the desired number of features remains. Wrapper methods account for feature interactions and are tailored to the specific model being used, but they are computationally expensive since they require many rounds of model training. Embedded methods build feature selection directly into the model training process itself — L1 (Lasso) regularization is a classic example, as it drives the coefficients of unhelpful features to exactly zero during training, and tree-based models naturally produce feature importance scores (e.g., based on how much a feature reduces impurity) as a side effect of fitting.
Cricket analogy: A coach trials different opening pairs in real matches, dropping the weakest pair each time until the best remains — thorough but costly in practice games, unlike a scorecard where a player's low strike-rate contribution simply falls out automatically from the season stats.
from sklearn.datasets import load_breast_cancer
from sklearn.feature_selection import SelectKBest, f_classif, RFE
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
# Filter method: keep top 10 features by ANOVA F-statistic
selector = SelectKBest(score_func=f_classif, k=10)
X_filtered = selector.fit_transform(X, y)
print('filter method selected', X_filtered.shape[1], 'features')
# Wrapper method: Recursive Feature Elimination down to 10 features
model = LogisticRegression(max_iter=5000)
rfe = RFE(estimator=model, n_features_to_select=10)
rfe.fit(X, y)
print('RFE selected features:', rfe.support_.sum())
# Embedded method: Lasso zeroes out coefficients of unhelpful features
from sklearn.linear_model import LassoCV
lasso = LassoCV(cv=5).fit(X, y)
print('non-zero coefficients:', (lasso.coef_ != 0).sum())Feature selection and dimensionality reduction (like PCA) both shrink the feature space, but they work differently: feature selection keeps a subset of the *original* features unchanged (so results remain interpretable), whereas PCA creates new, transformed features that are linear combinations of the originals and are harder to interpret directly.
If you perform feature selection using the entire dataset (including what will become your test set) before splitting, you leak information about the test set into your choice of features — the correct practice is to perform feature selection inside each training fold (or on the training split only), just like scaling.
- Feature selection keeps only the most useful features, reducing overfitting risk and improving interpretability and speed.
- Filter methods score features independently of a model using statistical tests (correlation, chi-squared, mutual information) — fast but ignore interactions.
- Wrapper methods (e.g., Recursive Feature Elimination) evaluate feature subsets by training real models — accurate but computationally expensive.
- Embedded methods build selection into training itself, e.g., Lasso's L1 penalty zeroing out coefficients, or tree-based feature importances.
- Feature selection preserves original, interpretable features, unlike PCA which creates new transformed components.
- Feature selection must be performed within training folds only to avoid leaking test information into the choice of features.
Practice what you learned
1. What distinguishes a filter method from a wrapper method for feature selection?
2. Which of the following is an example of an embedded feature selection method?
3. What is a key weakness of filter methods for feature selection?
4. How does feature selection differ from PCA-based dimensionality reduction?
5. Why must feature selection be performed only on the training fold, not the full dataset before splitting?
Was this page helpful?
You May Also Like
Dimensionality Reduction and PCA
See how Principal Component Analysis compresses many correlated features into fewer uncorrelated components while preserving most of the variance in data.
What Is Feature Engineering?
Explore how transforming raw data into informative input variables often matters more for model quality than the choice of algorithm itself.
Overfitting and Underfitting
The two failure modes of model fitting — memorizing noise in training data versus failing to capture real patterns — and the diagnostic and mitigation techniques for each.
Regularization: Ridge and Lasso
Explains how L2 (ridge) and L1 (lasso) penalties shrink regression coefficients to combat overfitting, and why lasso can perform feature selection.