Feature Selection
Feature selection is the process of choosing a subset of the most relevant input variables from a dataset to use in building a machine learning model, discarding redundant or uninformative features.
Definition
Feature selection is the process of choosing a subset of the most relevant input variables from a dataset to use in building a machine learning model, discarding redundant or uninformative features.
Overview
Real-world datasets often contain far more candidate features than are actually useful for predicting a target variable — some are irrelevant, some are redundant with other features, and some may even introduce noise that hurts model performance. Feature selection addresses this by systematically identifying and retaining only the most predictive subset of features, which is distinct from dimensionality reduction techniques like PCA that transform features into new derived dimensions rather than selecting from the original ones. Feature selection methods generally fall into three categories. Filter methods score each feature independently using a statistical measure — correlation with the target, chi-squared tests, or mutual information — and select the top-scoring features without involving a specific model, making them fast but blind to feature interactions. Wrapper methods, such as recursive feature elimination, repeatedly train and evaluate a model on different feature subsets to directly optimize for that model's performance, which is more accurate but computationally expensive. Embedded methods integrate feature selection into the model training process itself, such as L1 (Lasso) regularization, which drives the coefficients of unimportant features toward zero as a side effect of training. Good feature selection improves model interpretability, reduces training time and computational cost, mitigates overfitting caused by high-dimensional, noisy data (the "curse of dimensionality"), and can improve generalization to new data by removing spurious correlations. It remains a core preprocessing step in traditional machine learning pipelines involving tabular data, though it is less emphasized in deep learning contexts where models like neural networks can often learn to weight relevant features automatically from raw or lightly processed input, especially with abundant training data.
Key Concepts
- Selects a subset of the most predictive original input features
- Distinct from dimensionality reduction, which creates new transformed features
- Filter methods score features independently using statistical measures
- Wrapper methods evaluate feature subsets using actual model performance
- Embedded methods integrate selection into model training, like L1 regularization
- Reduces overfitting risk from high-dimensional or noisy data
- Improves model interpretability and training efficiency
- A core preprocessing step in traditional tabular machine learning pipelines
Use Cases
Frequently Asked Questions
From the Blog
How to A/B Test an LLM Feature With Real Users
Judge model changes on behaviour, not on offline scores. Pick one primary behavioural metric, define guardrails that stop the experiment automatically, randomise at the unit users actually experience, and hold the test long enough for the slow signals — retention and follow-up rate — to arrive.
Read More AI & TechnologyBuilding a golden evaluation set for an LLM feature
Your golden set makes every later decision measurable. Learn to mine real inputs and past failures, size and stratify it, and keep it honest as the product changes.
Read More AI & TechnologyStaged rollout for an LLM feature: flags, cohorts and rollback
You cannot fully validate an LLM feature pre-launch, so the rollout is the test. Design flags, cohorts, guardrail metrics and rollback triggers before you ship.
Read More Data ScienceFeature Engineering: Turning Data Into Signal
Feature engineering turns raw columns into inputs a model can actually learn from. Learn the core techniques that often matter more than the algorithm itself.
Read More