Wrapper Methods — Recursive Feature Elimination — Data Analysis & Feature Engineering

Wrapper methods select features by repeatedly training a model and evaluating which features most improve its performance, capturing interactions and complementarity that filter methods entirely miss. Recursive Feature Elimination is the most widely used wrapper approach: it trains a model, scores each feature by its importance to that model, eliminates the weakest features, and repeats until the desired number remain. It exists because the relevance of a feature is ultimately defined by whether it helps a specific model predict better, not by its individual statistical properties, and only training the model on various feature subsets can reveal this. Wrapper methods are more powerful than filters but more expensive, so the practical workflow pairs a cheap filter pass to shrink the space with a subsequent RFE pass for precision — the two-stage approach that balances cost and quality.

Analogy🏏Cricket

🏏 Think of it like cricket: Reducing twenty batting statistics to two dimensions with PCA produces a flat projection that may look like one undifferentiated cloud, while t-SNE or UMAP finds that the data actually organises into distinct clusters — aggressive pinch-hitters, steady anchors, explosive finishers — whose separation PCA's flat projection smeared together. Just as the non-linear reduction reveals the natural groupings that flat projection could not, t-SNE and UMAP reveal structure that PCA misses because it can only flatten, not curve to follow the data's natural shape. The insight is that non-linear reduction methods follow the data's true curved geometry rather than forcing a flat projection, revealing the cluster structure and local neighbourhoods that linear methods like PCA cannot preserve.