Regularization (L1/L2) Cheat Sheet
Covers L1 (Lasso) and L2 (Ridge) regularization for linear models, including Elastic Net, scikit-learn code, and hyperparameter tuning tips.
2 PagesIntermediateMar 5, 2026
Ridge & Lasso in scikit-learn
Fit L2 and L1 regularized linear models with proper feature scaling.
python
from sklearn.linear_model import Ridge, Lassofrom sklearn.preprocessing import StandardScaler# Always scale features before regularizationscaler = StandardScaler()X_train_scaled = scaler.fit_transform(X_train)X_test_scaled = scaler.transform(X_test)# Ridge (L2) - shrinks coefficients toward zeroridge = Ridge(alpha=1.0) # alpha = lambda, higher = more regularizationridge.fit(X_train_scaled, y_train)# Lasso (L1) - can shrink coefficients to exactly zero (feature selection)lasso = Lasso(alpha=0.1)lasso.fit(X_train_scaled, y_train)print(lasso.coef_) # some coefficients will be 0.0
Elastic Net & Logistic Regression
Combine L1/L2 penalties and apply regularization to classification models.
python
from sklearn.linear_model import ElasticNet, ElasticNetCV, LogisticRegression# Elastic Net combines L1 and L2 penalties# l1_ratio=1 -> pure Lasso, l1_ratio=0 -> pure Ridgeen = ElasticNet(alpha=0.1, l1_ratio=0.5)en.fit(X_train_scaled, y_train)# Cross-validated search over alpha and l1_ratioen_cv = ElasticNetCV(l1_ratio=[.1, .5, .7, .9, .95, 1], cv=5)en_cv.fit(X_train_scaled, y_train)# Regularized logistic regression (classification)# penalty: 'l1', 'l2', 'elasticnet', None# C = 1 / lambda -> smaller C = stronger regularizationclf_l2 = LogisticRegression(penalty='l2', C=1.0, solver='lbfgs')clf_l1 = LogisticRegression(penalty='l1', C=0.5, solver='liblinear')
Core Concepts
The math and intuition behind L1 and L2 penalties.
- L1 penalty (Lasso)- Adds λΣ|wᵢ| to the loss function; produces sparse solutions by driving some weights exactly to 0
- L2 penalty (Ridge)- Adds λΣwᵢ² to the loss function; shrinks all weights smoothly toward 0 but rarely to exactly 0
- Elastic Net- Combines L1 and L2: λ₁Σ|wᵢ| + λ₂Σwᵢ²; useful when features are correlated
- alpha / lambda (λ)- Regularization strength; higher values increase bias and reduce variance
- C (scikit-learn)- Inverse of regularization strength in LogisticRegression/SVC; smaller C means a stronger penalty
- Bias-variance tradeoff- Regularization increases bias but reduces variance, often lowering test error
- Feature scaling- Required before regularization since penalty magnitude depends on coefficient scale
Hyperparameter Tuning
Practical guidance for choosing regularization strength.
- GridSearchCV- Search alpha over a log scale, e.g. np.logspace(-4, 4, 50)
- RidgeCV / LassoCV- Built-in cross-validated estimators that select alpha automatically
- Standardization- Use StandardScaler so all coefficients are penalized on the same scale
- Sparse solutions- Use Lasso/L1 when you expect only a subset of features to matter
- Multicollinearity- Ridge handles correlated features better than Lasso, which picks one arbitrarily
Weight Decay in Neural Nets: SGD vs Adam vs AdamW
L2 regularization and 'weight decay' are only mathematically equivalent for plain SGD — Adam breaks that equivalence.
python
import torch.optim as optim# With plain SGD, adding L2 penalty to the loss (lambda * ||w||^2) and# subtracting lambda * w directly from the weights each step are equivalent.sgd = optim.SGD(model.parameters(), lr=0.01, weight_decay=1e-4)# With Adam, weight_decay implemented as an L2 term gets divided by the# adaptive per-parameter second-moment estimate, so it's no longer a# clean, uniform shrinkage -- large-gradient parameters get under-regularized.adam_coupled = optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4) # biased# AdamW decouples weight decay from the gradient-based update entirely:# w = w - lr * (adam_update + weight_decay * w)# This restores the intended uniform shrinkage and is the standard choice# for training transformers.adamw = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
Proximal Gradient Descent (ISTA) for L1
L1's non-differentiability at zero means plain gradient descent can't be used directly; the soft-thresholding operator solves it.
python
import numpy as npdef soft_threshold(x, thresh): # Proximal operator of the L1 norm: shrinks values toward 0 and # zeroes out anything within [-thresh, thresh] return np.sign(x) * np.maximum(np.abs(x) - thresh, 0.0)def ista_lasso(X, y, alpha=0.1, lr=None, n_iters=500): n, p = X.shape if lr is None: lr = 1.0 / np.linalg.norm(X, 2) ** 2 # step <= 1/L, L = largest eigenvalue of X^T X w = np.zeros(p) for _ in range(n_iters): grad = X.T @ (X @ w - y) / n # gradient of the smooth MSE term only w = soft_threshold(w - lr * grad, alpha * lr) # proximal step handles the L1 term return w
Regularization Path with LARS
Compute the full Lasso solution path across all alpha values in one pass instead of refitting per alpha.
python
from sklearn.linear_model import lars_pathimport matplotlib.pyplot as plt# LARS (Least Angle Regression) computes the exact piecewise-linear path# of Lasso coefficients as alpha decreases from infinity to 0, in roughly# the same cost as a single OLS fit -- far cheaper than grid-searching alpha.alphas, active, coefs = lars_path(X_train_scaled, y_train, method="lasso")for i in range(coefs.shape[0]): plt.plot(alphas, coefs[i], label=f"feature {i}")plt.xscale("log")plt.xlabel("alpha (log scale)")plt.ylabel("coefficient value")plt.title("Lasso regularization path")# Reading the plot: features whose coefficient lines hit 0 first (at the# largest alpha) are the least important under L1 selection
Group Lasso & Streaming Elastic Net
Regularize predefined groups of features together, and fit elastic-net-penalized models on data too large to fit in memory.
python
from sklearn.linear_model import SGDClassifierfrom group_lasso import GroupLasso # pip install group-lasso# Group Lasso: penalizes the L2 norm of each feature group, zeroing out# entire groups (e.g. all one-hot columns from a single categorical) together# rather than individual dummy columnsgl = GroupLasso( groups=group_ids, # array mapping each column to its group id group_reg=0.05, l1_reg=0.0, supress_warning=True,)gl.fit(X_train_scaled, y_train)# SGDClassifier scales elastic-net-penalized linear models to datasets that# don't fit in memory via mini-batch / partial_fit updatessgd_en = SGDClassifier( loss="log_loss", penalty="elasticnet", alpha=1e-4, l1_ratio=0.15, max_iter=1000, tol=1e-3,)for X_batch, y_batch in stream_batches(): sgd_en.partial_fit(X_batch, y_batch, classes=[0, 1])
Theoretical Foundations
The statistical and optimization theory underlying L1/L2, beyond the sklearn API.
- Bayesian interpretation- L2 regularization corresponds to a Gaussian prior on the weights (MAP estimation); L1 corresponds to a Laplace prior, whose sharp peak at 0 is why it induces sparsity
- Why L1 gives sparsity, L2 doesn't- the L1 ball has corners on the coordinate axes, so the loss contour is likely to first touch it exactly at a corner (a zero coefficient); the L2 ball is smooth, so the tangent point rarely lands on an axis
- L0 penalty- directly penalizes the count of nonzero weights; NP-hard to optimize exactly, which is why L1 is used as its convex relaxation
- Implicit regularization- early stopping, dropout, and even SGD's inherent noise all act as regularizers without an explicit penalty term in the loss
- Ridge closed-form solution- w = (XᵀX + λI)⁻¹Xᵀy; adding λI to XᵀX before inverting also stabilizes the solution when features are collinear or n < p
- Effective degrees of freedom- for Ridge, df(λ) = Σ dᵢ²/(dᵢ²+λ) using the singular values dᵢ of X; quantifies how much regularization shrinks model complexity as a continuous (not integer) quantity
- Double descent- in modern over-parameterized models, test error can decrease again past the interpolation threshold even without explicit regularization, complicating the classic bias-variance-driven tuning intuition
Pro Tip
When features are highly correlated, prefer Elastic Net over pure Lasso - Lasso tends to arbitrarily select one feature from a correlated group and zero out the rest, which hurts interpretability and stability.
Was this cheat sheet helpful?
Explore Topics
#RegularizationL1L2#RegularizationL1L2CheatSheet#DataScience#Intermediate#Ridge#Lasso#Scikit#Learn#MachineLearning#CheatSheet#SkillVeris
