What is the difference between L1 and L2 regularization?
Understand the difference between L1 (Lasso) and L2 (Ridge) regularization, why L1 gives sparse models, and how each fights overfitting in machine learning.
Expected Interview Answer
L1 (Lasso) regularization adds the sum of the absolute values of the weights to the loss, driving some weights exactly to zero and performing feature selection, while L2 (Ridge) adds the sum of squared weights, shrinking them smoothly toward but rarely exactly to zero.
Both techniques penalise large coefficients to reduce overfitting, but the shape of the penalty changes the outcome. L1's absolute-value penalty has sharp corners at zero, so the optimum often lands on an axis where a weight becomes exactly zero, yielding sparse, interpretable models. L2's squared penalty is smooth and rounded, spreading shrinkage across all weights and handling correlated features more gracefully. Elastic Net combines the two to get sparsity and stability together.
- Both reduce overfitting by penalising large weights
- L1 produces sparse models and automatic feature selection
- L2 handles correlated features and stabilises coefficients
- L2 is differentiable everywhere, easing optimisation
- Elastic Net blends L1 and L2 for the best of both
AI Mentor Explanation
A captain trimming a bloated squad can act two ways. L1 is ruthless selection: fringe players are cut to zero and dropped entirely, leaving a lean, clear first eleven. L2 is a fitness cap that trims every player's workload a little so no one is overused, but keeps the whole squad on the books. One deletes marginal contributors, the other tones everyone down while retaining them all.
Step-by-Step Explanation
Step 1
Start from the base loss
Take the model's usual loss such as mean squared error or cross-entropy.
Step 2
Add the penalty term
L1 adds lambda times the sum of |weights|; L2 adds lambda times the sum of weights squared.
Step 3
Understand the penalty shape
L1's diamond-shaped constraint has corners on the axes; L2's is a smooth circle.
Step 4
See the effect on weights
L1 pushes some weights exactly to zero (sparsity); L2 shrinks all weights smoothly toward zero.
Step 5
Tune lambda
Use cross-validation to pick the regularisation strength that best balances bias and variance.
What Interviewer Expects
- Correct definitions of the L1 and L2 penalty terms
- Why L1 yields sparsity and L2 does not
- Geometric intuition of the constraint shapes
- How each handles correlated features
- Awareness of Elastic Net as a combination
Common Mistakes
- Claiming L2 zeroes out weights like L1
- Forgetting to scale features before regularising
- Saying regularisation always improves accuracy
- Confusing the regularisation strength lambda's direction of effect
- Not knowing L1 is non-differentiable at zero
Best Answer (HR Friendly)
“L1 and L2 regularisation both discourage a model from relying too heavily on any one feature, which helps prevent overfitting. L1 can switch some features off entirely, giving a simpler model, while L2 gently shrinks all of them to keep things stable.”
Code Example
from sklearn.linear_model import Lasso, Ridge
lasso = Lasso(alpha=0.1) # L1: drives some coefficients to zero
ridge = Ridge(alpha=0.1) # L2: shrinks all coefficients smoothly
lasso.fit(X_train, y_train)
ridge.fit(X_train, y_train)
print('L1 non-zero coefs:', (lasso.coef_ != 0).sum())
print('L2 non-zero coefs:', (ridge.coef_ != 0).sum())Follow-up Questions
- Why does the L1 penalty produce exactly-zero weights while L2 does not?
- What is Elastic Net and when would you use it?
- How does increasing lambda affect bias and variance?
- Why must features be standardised before applying regularisation?
MCQ Practice
1. Which regularisation technique performs automatic feature selection?
L1's absolute-value penalty drives some coefficients exactly to zero, effectively removing those features from the model.
2. What penalty term does L2 regularisation add to the loss?
L2 (Ridge) adds lambda times the sum of the squared weights, shrinking all coefficients smoothly toward zero.
3. Which method combines L1 and L2 penalties?
Elastic Net blends the L1 and L2 penalties, delivering sparsity while handling correlated features more stably than pure Lasso.
Flash Cards
L1 penalty term? — Lambda times the sum of the absolute values of the weights.
L2 penalty term? — Lambda times the sum of the squared weights.
Which gives sparsity? — L1 (Lasso) — it drives some weights exactly to zero.
Which handles correlated features better? — L2 (Ridge) — it shrinks correlated weights together smoothly.
What is Elastic Net? — A combination of L1 and L2 penalties for sparsity plus stability.