Learning rate schedules and warm restarts — Deep Learning & Neural Networks

The learning rate is the single most important hyperparameter in neural network training — more impactful than batch size, architecture depth, or optimiser choice. A fixed learning rate, however optimal it seems, is a poor strategy: too large early in training and the model overshoots minima; too small late in training and it crawls to convergence. Learning rate schedules solve this by varying the learning rate over time according to a predefined or adaptive rule. Warm-up phases start with a very small learning rate and gradually increase it, preventing catastrophic early updates when weights are randomly initialised. Decay phases progressively reduce the learning rate, allowing fine-grained convergence to the loss minimum. Cosine annealing creates smooth oscillatory decay that avoids sharp transitions. Warm restarts periodically reset the learning rate to its initial value, helping the model escape local minima and explore different regions of the loss landscape. Without scheduling, models plateau early and leave substantial accuracy on the table.

Analogy🏏Cricket

🏏 Think of it like cricket: The Duckworth-Lewis-Stern (DLS) method, the ICC bowling economy metric, and the net run rate each measure team performance but optimise for completely different things — DLS cares about wickets-in-hand, economy cares about runs-per-over, NRR cares about tournament-wide margins. Using the wrong metric to evaluate a bowler is like using MSE for classification: technically computable but optimising the wrong thing entirely. Just as you would never rank a spinner by batting average, you should never use MSE when your model predicts probabilities. Just as focal DLS adjustments down-weight easy chases and amplify close finishes, focal loss down-weights easy examples and amplifies hard minority-class examples. The choice of scoring metric defines what excellence means — and so does the choice of loss function.