What is Regularization (L1 and L2) in ML?
Understand L1 (Lasso) and L2 (Ridge) regularization: how each fights overfitting, when to use sparsity vs shrinkage, and how to tune lambda in scikit-learn.
Expected Interview Answer
Regularization adds a penalty on model coefficient size to the loss function so the model stays simpler and generalizes better, directly combating overfitting. L1 (Lasso) penalizes the sum of absolute weights and can zero some out; L2 (Ridge) penalizes the sum of squared weights and shrinks them smoothly.
During training the optimizer minimizes prediction error plus lambda times a penalty on the weights, trading a little training accuracy for lower variance. L1's diamond-shaped constraint pushes many coefficients exactly to zero, giving automatic feature selection and sparse models, while L2's circular constraint shrinks all coefficients toward zero without eliminating them, which handles correlated features more gracefully. Elastic Net blends both penalties, and the strength lambda is tuned by cross-validation.
- Reduces overfitting and improves test generalization
- L1 performs automatic feature selection via sparsity
- L2 stabilizes models with correlated or many features
- Controls model complexity through a single tunable lambda
- Improves numerical stability of the fitted weights
AI Mentor Explanation
A batter who tries every flamboyant shot memorizes each specific delivery but crumbles against new bowling. A coach imposing a penalty for reckless strokes forces a compact, repeatable technique. Regularization is that penalty on the model: it discourages wild, oversized coefficients so the batter's method generalizes to unseen deliveries instead of overfitting past balls.
Step-by-Step Explanation
Step 1
Start from the loss
Begin with the base loss, such as mean squared error, that measures prediction error on the training data.
Step 2
Add a penalty term
Append lambda times a norm of the weights: absolute values for L1, squared values for L2.
Step 3
Choose lambda
Tune the regularization strength lambda with cross-validation; larger lambda means simpler, more shrunken models.
Step 4
Optimize the combined objective
Minimize error plus penalty so the optimizer balances fitting the data against keeping weights small.
Step 5
Inspect the result
L1 yields sparse weights with feature selection; L2 yields small, non-zero weights that handle correlation.
What Interviewer Expects
- Clear distinction between L1 sparsity and L2 shrinkage
- Understanding of the bias-variance tradeoff
- Role of the lambda hyperparameter and how to tune it
- Why L1 zeros coefficients (geometry of the constraint)
- Awareness of Elastic Net combining both penalties
Common Mistakes
- Claiming L2 produces sparse, zeroed coefficients (that is L1)
- Forgetting to scale features before applying regularization
- Setting lambda without cross-validation
- Confusing regularization with dropout or early stopping only
- Regularizing the intercept/bias term unintentionally
Best Answer (HR Friendly)
“Regularization is a way to stop a model from memorizing its training data by penalizing overly large settings. L1 can switch off unhelpful features entirely, while L2 gently shrinks all of them, and both help the model work better on new, unseen data.”
Code Example
from sklearn.linear_model import Lasso, Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score
# L1: drives some coefficients to exactly zero (feature selection)
lasso = make_pipeline(StandardScaler(), Lasso(alpha=0.1))
# L2: shrinks all coefficients smoothly toward zero
ridge = make_pipeline(StandardScaler(), Ridge(alpha=1.0))
for name, model in [('Lasso/L1', lasso), ('Ridge/L2', ridge)]:
scores = cross_val_score(model, X, y, cv=5, scoring='r2')
print(name, 'mean R2:', scores.mean())
lasso.fit(X, y)
print('Non-zero features:', (lasso[-1].coef_ != 0).sum())Follow-up Questions
- Why does L1 produce sparse solutions but L2 does not?
- What is Elastic Net and when would you use it?
- How does regularization relate to the bias-variance tradeoff?
- Why must features be scaled before regularizing?
- How would you select the best lambda in practice?
MCQ Practice
1. Which regularization technique can set some coefficients exactly to zero?
L1's absolute-value penalty has a geometry that pushes coefficients to exactly zero, giving automatic feature selection; L2 only shrinks them.
2. Increasing lambda in regularization generally does what?
A larger lambda applies a stronger penalty on weight magnitude, shrinking coefficients and producing a simpler, lower-variance model.
3. L2 regularization is especially helpful when features are:
Ridge/L2 shrinks correlated coefficients together and distributes weight among them, giving more stable estimates than an unregularized fit.
Flash Cards
What does L1 regularization do to coefficients? — It penalizes the sum of absolute weights and can drive some to exactly zero, performing automatic feature selection (Lasso).
What does L2 regularization do to coefficients? — It penalizes the sum of squared weights, smoothly shrinking all of them toward zero without eliminating any (Ridge).
What controls regularization strength? — The hyperparameter lambda (alpha in scikit-learn); larger values mean a stronger penalty and simpler models, tuned via cross-validation.
What is Elastic Net? — A regularizer combining L1 and L2 penalties, giving both sparsity and stability for correlated features.
Why scale features before regularizing? — Penalties depend on weight magnitude, so unscaled features get penalized unequally; standardizing puts all features on comparable scales.