Optimisers — SGD, Adam, RMSProp — Deep Learning & Neural Networks

Once a loss function produces a gradient, an optimiser decides how to use that gradient to update weights. This decision is far more consequential than it appears: the same gradient can lead to fast, stable convergence or slow, oscillating divergence depending purely on how the optimiser transforms it into a weight update. Stochastic Gradient Descent (SGD) is the original and simplest optimiser — it applies the gradient directly, scaled by a learning rate. But vanilla SGD struggles with saddle points, sparse gradients, and different learning rate needs across different parameters. RMSProp divides each gradient by a running average of its recent magnitudes, automatically adapting the step size per parameter. Adam combines momentum (memory of past gradients) with RMSProp-style adaptive step sizes, making it the most widely used optimiser in deep learning. Choosing the wrong optimiser does not just slow training — it can prevent convergence entirely, especially in deep networks where gradient landscapes are complex and highly non-convex.

Analogy🏏Cricket

🏏 Think of it like cricket: The Duckworth-Lewis-Stern (DLS) method, the ICC bowling economy metric, and the net run rate each measure team performance but optimise for completely different things — DLS cares about wickets-in-hand, economy cares about runs-per-over, NRR cares about tournament-wide margins. Using the wrong metric to evaluate a bowler is like using MSE for classification: technically computable but optimising the wrong thing entirely. Just as you would never rank a spinner by batting average, you should never use MSE when your model predicts probabilities. Just as focal DLS adjustments down-weight easy chases and amplify close finishes, focal loss down-weights easy examples and amplifies hard minority-class examples. The choice of scoring metric defines what excellence means — and so does the choice of loss function.