Seq2Seq and attention mechanism intro — Deep Learning & Neural Networks

Sequence-to-sequence (seq2seq) models map an input sequence of variable length to an output sequence of variable length — translating a sentence, summarising a match report, generating a commentary response from ball events. The encoder-decoder architecture encodes the entire input sequence into a fixed-length context vector (the encoder's final hidden state), which the decoder then uses to generate the output sequence step by step. The critical bottleneck is that all input information must pass through this single fixed-length vector — for long sequences, the context vector cannot hold all relevant information, causing performance to degrade sharply as sequence length grows. The attention mechanism, introduced by Bahdanau et al. in 2015, solves this bottleneck by allowing the decoder to dynamically attend to different parts of the encoder's output at each decoding step — instead of a single context vector, the decoder can access the entire encoder hidden state sequence, weighting each position by how relevant it is to the current decoding step. Attention is the foundational idea behind the transformer architecture — understanding encoder-decoder seq2seq with attention is the essential prerequisite for understanding BERT, GPT, and every modern large language model.

Analogy🏏Cricket

🏏 Think of it like cricket: The Duckworth-Lewis-Stern (DLS) method, the ICC bowling economy metric, and the net run rate each measure team performance but optimise for completely different things — DLS cares about wickets-in-hand, economy cares about runs-per-over, NRR cares about tournament-wide margins. Using the wrong metric to evaluate a bowler is like using MSE for classification: technically computable but optimising the wrong thing entirely. Just as you would never rank a spinner by batting average, you should never use MSE when your model predicts probabilities. Just as focal DLS adjustments down-weight easy chases and amplify close finishes, focal loss down-weights easy examples and amplifies hard minority-class examples. The choice of scoring metric defines what excellence means — and so does the choice of loss function.