Learning Rate Scheduling
Learning rate scheduling is the practice of systematically adjusting a model's learning rate during training, rather than keeping it fixed, to speed up convergence and improve final model quality.
Definition
Learning rate scheduling is the practice of systematically adjusting a model's learning rate during training, rather than keeping it fixed, to speed up convergence and improve final model quality.
Overview
The learning rate controls how large a step an optimizer takes when updating model parameters in the direction of the negative gradient. A learning rate that is too high can cause training to diverge or oscillate; one that is too low converges slowly and can get stuck in poor local minima or saddle points. Learning rate scheduling addresses this by varying the learning rate over the course of training according to a predefined or adaptive rule. Common schedules include step decay (dropping the learning rate by a fixed factor every set number of epochs), exponential decay, cosine annealing (smoothly decreasing the learning rate following a cosine curve, often down to near zero by the end of training), and linear decay. Many training regimes, particularly for large transformer models, also use a warmup phase at the start of training, during which the learning rate increases linearly from a small value up to its peak before decay begins; warmup helps stabilize training early on, when gradients and parameter estimates are least reliable, especially with adaptive optimizers like Adam. Adaptive schedules such as "reduce on plateau" instead monitor a validation metric and lower the learning rate only when progress stalls, rather than following a fixed schedule. Cyclical learning rate schedules oscillate the rate between bounds to help escape sharp local minima. Learning rate scheduling is one of the most impactful and cheapest hyperparameter choices in deep learning — it requires no architectural change, yet the right schedule can meaningfully improve final accuracy and reduce total training time, which is why virtually every modern large-scale training run (from image classifiers to large language models) uses some form of warmup-plus-decay schedule.
Key Concepts
- Adjusts the learning rate over the course of training rather than keeping it fixed
- Warmup phase gradually increases the learning rate at the start of training
- Cosine annealing smoothly decays the rate following a cosine curve
- Step decay reduces the rate by a fixed factor at set intervals
- Reduce-on-plateau adapts the schedule based on validation performance
- Cyclical schedules oscillate the rate to help escape sharp minima
- Standard practice in training large transformer-based models
- Requires no architectural changes, only optimizer configuration
Use Cases
Frequently Asked Questions
From the Blog
What Is a Learning Rate and How to Tune It
The learning rate controls how big a step a model takes when updating its weights during training — the single most important hyperparameter to get right.
Read More AI & TechnologyWhat Is CTR? Click-Through Rate Explained
CTR, or click-through rate, measures the percentage of people who click a link, ad, or search result out of everyone who saw it. This guide explains how it's calculated, what affects it, and how to improve it.
Read More AI & TechnologyHow to Choose Learning Rate and Epochs for a LoRA Run
Start from a conservative configuration, run a short training pass, and let the loss curves tell you what to change. Rank, alpha, learning rate and epoch count interact, so the productive method is one variable at a time against a held-out set rather than a search over everything at once.
Read More AI & TechnologyHow to Measure Hallucination Rate Without Manual Review
You can score factuality automatically by splitting each answer into atomic claims and checking every claim against the retrieved source text. This article shows how to build that pipeline, how to calibrate its verdicts against a small human-labelled sample, and where it quietly fails.
Read More