Embedded Methods — Lasso and Tree Importance — Data Analysis & Feature Engineering

Embedded methods perform feature selection as an integral part of model training, combining the accuracy of wrapper methods with a computational cost far closer to fitting a single model. They exist as the practical sweet spot between slow wrappers and blind filters: Lasso regression drives irrelevant feature coefficients to exactly zero through L1 regularisation, and tree-based importance measures accumulate how much each feature improves splits across the ensemble, both selecting features without a separate iterative loop. These methods are among the most widely used in production because they deliver model-integrated selection efficiently, and they have the additional advantage that the selected features are precisely those the model actually uses, making the selection and model internally consistent. Understanding both Lasso shrinkage and tree importance — including their biases and limitations — is essential for principled embedded selection.

Analogy🏏Cricket

🏏 Think of it like cricket: Reducing twenty batting statistics to two dimensions with PCA produces a flat projection that may look like one undifferentiated cloud, while t-SNE or UMAP finds that the data actually organises into distinct clusters — aggressive pinch-hitters, steady anchors, explosive finishers — whose separation PCA's flat projection smeared together. Just as the non-linear reduction reveals the natural groupings that flat projection could not, t-SNE and UMAP reveal structure that PCA misses because it can only flatten, not curve to follow the data's natural shape. The insight is that non-linear reduction methods follow the data's true curved geometry rather than forcing a flat projection, revealing the cluster structure and local neighbourhoods that linear methods like PCA cannot preserve.