Regularisation — Dropout, BatchNorm, L2 — Deep Learning & Neural Networks

Overfitting is the central failure mode of deep neural networks: a model with millions of parameters can memorise training examples rather than learning the underlying patterns, achieving near-perfect training accuracy while performing poorly on unseen data. Regularisation is the collection of techniques that constrain the model's capacity or add noise during training to prevent memorisation and improve generalisation. L2 regularisation penalises large weights directly in the loss function, keeping the model's parameter space compact. Dropout randomly deactivates neurons during each training forward pass, preventing any single neuron from becoming a critical node that the model overly relies upon. Batch Normalisation (BatchNorm) normalises layer activations across each mini-batch, stabilising training and acting as a mild regulariser. Each technique addresses overfitting from a different angle, and all three are used simultaneously in most production architectures. Without at least one form of regularisation, any model with sufficient capacity will overfit given enough training epochs.

Analogy🏏Cricket

🏏 Think of it like cricket: The Duckworth-Lewis-Stern (DLS) method, the ICC bowling economy metric, and the net run rate each measure team performance but optimise for completely different things — DLS cares about wickets-in-hand, economy cares about runs-per-over, NRR cares about tournament-wide margins. Using the wrong metric to evaluate a bowler is like using MSE for classification: technically computable but optimising the wrong thing entirely. Just as you would never rank a spinner by batting average, you should never use MSE when your model predicts probabilities. Just as focal DLS adjustments down-weight easy chases and amplify close finishes, focal loss down-weights easy examples and amplifies hard minority-class examples. The choice of scoring metric defines what excellence means — and so does the choice of loss function.