Weight Decay
Weight decay is a regularization technique that adds a penalty proportional to the magnitude of a model's weights to the loss function, discouraging overly large parameter values and reducing overfitting.
Definition
Weight decay is a regularization technique that adds a penalty proportional to the magnitude of a model's weights to the loss function, discouraging overly large parameter values and reducing overfitting.
Overview
Neural networks and other parametric models can fit training data very closely by growing certain weights to large magnitudes, which often signals overfitting to noise or idiosyncrasies in the training set rather than learning generalizable patterns. Weight decay counteracts this by adding a term to the loss — typically the sum of squared weights (L2 regularization) scaled by a small coefficient — so that the optimizer is pushed to keep weights small unless a larger value meaningfully improves the model's fit to the data. In practice, weight decay is usually implemented in one of two mathematically related but not identical ways: as an explicit L2 penalty added directly to the loss function before computing gradients, or as a direct multiplicative shrinkage of weights applied at each optimizer step, independent of the loss gradient. This second form, known as decoupled weight decay, is used in the popular AdamW optimizer and was introduced because combining standard L2 regularization with adaptive optimizers like Adam does not behave the way it does with plain stochastic gradient descent — the adaptive learning rate scaling interferes with the regularization effect. AdamW decouples the two, applying weight decay independently of the gradient-based update, which was shown to improve generalization. The strength of weight decay is controlled by a coefficient, commonly denoted lambda, that must be tuned: too small and it has negligible regularizing effect, too large and it can underfit by shrinking useful weights toward zero. Typical values range from very small (like 0.01) for large language model training to somewhat larger values for smaller models, and it is common to exclude bias terms and normalization layer parameters from weight decay, since penalizing those can hurt performance without providing meaningful regularization. Weight decay is one of the most widely used and computationally cheap regularization techniques, and it is standard in training recipes for image classifiers, transformers, and virtually all modern deep learning models, usually applied alongside other regularization methods like dropout and early stopping.
Key Concepts
- Penalizes large weight magnitudes to reduce overfitting
- Classically implemented as an L2 penalty added to the loss function
- Decoupled weight decay (as in AdamW) applies shrinkage independent of gradients
- Controlled by a tunable coefficient (lambda) balancing fit versus regularization
- Commonly excludes bias terms and normalization parameters from the penalty
- Computationally cheap compared to other regularization techniques
- Standard component of training recipes for transformers and CNNs alike
- Often used together with dropout, data augmentation, and early stopping