RNN and vanishing gradient problem — Deep Learning & Neural Networks

Dense networks and CNNs treat every input independently — a Dense layer processing word 50 in a sentence has no memory of words 1 through 49. This is catastrophically wrong for sequential data. The meaning of 'played a brilliant shot' depends entirely on the preceding context — was it 'Rohit played a brilliant shot' or 'the bowler played a brilliant shot into Rohit's hands'? Recurrent Neural Networks (RNNs) solve this by maintaining a hidden state — a vector that persists across time steps, accumulating context as the sequence is processed. At each step t, the hidden state h_t = tanh(W_h × h_{t-1} + W_x × x_t + b) combines the previous hidden state with the current input. This makes RNNs theoretically capable of learning from arbitrarily long sequences. In practice, however, RNNs suffer catastrophically from the vanishing gradient problem: gradients are backpropagated through time (BPTT), and multiplying through dozens of tanh derivatives (each ≤ 0.25) causes the gradient to collapse to near-zero before reaching the early time steps. The network learns short-term patterns but completely forgets anything more than 5–10 steps back. Understanding this failure mode is why LSTM and GRU exist, and why transformers replaced both.

Analogy🏏Cricket

🏏 Think of it like cricket: The Duckworth-Lewis-Stern (DLS) method, the ICC bowling economy metric, and the net run rate each measure team performance but optimise for completely different things — DLS cares about wickets-in-hand, economy cares about runs-per-over, NRR cares about tournament-wide margins. Using the wrong metric to evaluate a bowler is like using MSE for classification: technically computable but optimising the wrong thing entirely. Just as you would never rank a spinner by batting average, you should never use MSE when your model predicts probabilities. Just as focal DLS adjustments down-weight easy chases and amplify close finishes, focal loss down-weights easy examples and amplifies hard minority-class examples. The choice of scoring metric defines what excellence means — and so does the choice of loss function.