Loss functions — MSE, cross-entropy, focal — Deep Learning & Neural Networks

A neural network learns by minimising a loss function — a scalar number that measures how wrong its predictions are. Without a well-chosen loss function, gradient descent has no meaningful signal to follow, and training either diverges or converges to a useless solution. Mean Squared Error (MSE) penalises large prediction errors quadratically and is natural for regression tasks. Cross-Entropy loss is the information-theoretic measure of how well a predicted probability distribution matches the true distribution — it is the correct choice for classification because it directly optimises the predicted probabilities rather than raw logit values. Focal loss is a modern variant that solves a specific production problem: when 99% of training examples are easy negatives, standard cross-entropy spends most of its gradient budget on examples the model already handles well, starving difficult minority cases of learning signal. Understanding which loss to use — and why — is the difference between a model that trains and one that merely runs.

Analogy🏏Cricket

🏏 Think of it like cricket: The Duckworth-Lewis-Stern (DLS) method, the ICC bowling economy metric, and the net run rate each measure team performance but optimise for completely different things — DLS cares about wickets-in-hand, economy cares about runs-per-over, NRR cares about tournament-wide margins. Using the wrong metric to evaluate a bowler is like using MSE for classification: technically computable but optimising the wrong thing entirely. Just as you would never rank a spinner by batting average, you should never use MSE when your model predicts probabilities. Just as focal DLS adjustments down-weight easy chases and amplify close finishes, focal loss down-weights easy examples and amplifies hard minority-class examples. The choice of scoring metric defines what excellence means — and so does the choice of loss function.