Learning Rate Scheduling
Learning rate scheduling is the practice of systematically adjusting a model's learning rate during training, rather than keeping it fixed, to speed up convergence and improve final model quality.
Definition
Learning rate scheduling is the practice of systematically adjusting a model's learning rate during training, rather than keeping it fixed, to speed up convergence and improve final model quality.
Overview
The learning rate controls how large a step an optimizer takes when updating model parameters in the direction of the negative gradient. A learning rate that is too high can cause training to diverge or oscillate; one that is too low converges slowly and can get stuck in poor local minima or saddle points. Learning rate scheduling addresses this by varying the learning rate over the course of training according to a predefined or adaptive rule. Common schedules include step decay (dropping the learning rate by a fixed factor every set number of epochs), exponential decay, cosine annealing (smoothly decreasing the learning rate following a cosine curve, often down to near zero by the end of training), and linear decay. Many training regimes, particularly for large transformer models, also use a warmup phase at the start of training, during which the learning rate increases linearly from a small value up to its peak before decay begins; warmup helps stabilize training early on, when gradients and parameter estimates are least reliable, especially with adaptive optimizers like Adam. Adaptive schedules such as "reduce on plateau" instead monitor a validation metric and lower the learning rate only when progress stalls, rather than following a fixed schedule. Cyclical learning rate schedules oscillate the rate between bounds to help escape sharp local minima. Learning rate scheduling is one of the most impactful and cheapest hyperparameter choices in deep learning — it requires no architectural change, yet the right schedule can meaningfully improve final accuracy and reduce total training time, which is why virtually every modern large-scale training run (from image classifiers to large language models) uses some form of warmup-plus-decay schedule.
Key Concepts
- Adjusts the learning rate over the course of training rather than keeping it fixed
- Warmup phase gradually increases the learning rate at the start of training
- Cosine annealing smoothly decays the rate following a cosine curve
- Step decay reduces the rate by a fixed factor at set intervals
- Reduce-on-plateau adapts the schedule based on validation performance
- Cyclical schedules oscillate the rate to help escape sharp minima
- Standard practice in training large transformer-based models
- Requires no architectural changes, only optimizer configuration