Long Short-Term Memory (LSTM) networks, invented by Hochreiter and Schmidhuber in 1997, solved the vanishing gradient problem that made vanilla RNNs unable to learn from sequences longer than 10–20 steps. The key innovation was adding a cell state — a separate memory vector that runs through the entire sequence with only additive updates, providing a direct gradient highway analogous to ResNet's skip connections. Three learnable gates (forget, input, output) control what information is written to, retained in, and read from the cell state at each time step. The Gated Recurrent Unit (GRU), introduced by Cho et al. in 2014, is a simplified LSTM with two gates (reset and update) instead of three, sharing the cell state and hidden state into one vector. GRU achieves comparable accuracy to LSTM on most tasks while using 25% fewer parameters and training faster. Understanding the gating mechanism in LSTM and GRU is the foundation for understanding the attention mechanism in transformers — attention is essentially a fully learnable, input-dependent gating mechanism applied across all sequence positions simultaneously rather than sequentially.
35 minadvanced
LSTM and GRU — gating mechanisms
Analogy🏏Cricket
🏏 Think of it like cricket: The Duckworth-Lewis-Stern (DLS) method, the ICC bowling economy metric, and the net run rate each measure team performance but optimise for completely different things — DLS cares about wickets-in-hand, economy cares about runs-per-over, NRR cares about tournament-wide margins. Using the wrong metric to evaluate a bowler is like using MSE for classification: technically computable but optimising the wrong thing entirely. Just as you would never rank a spinner by batting average, you should never use MSE when your model predicts probabilities. Just as focal DLS adjustments down-weight easy chases and amplify close finishes, focal loss down-weights easy examples and amplifies hard minority-class examples. The choice of scoring metric defines what excellence means — and so does the choice of loss function.
Lesson 20 of 35
0% complete