Transformer architecture — self-attention

The Transformer, introduced in 'Attention Is All You Need' (Vaswani et al., 2017), replaced recurrent networks with a purely attention-based architecture and became the foundation of every major language model since. The key insight was radical: discard sequential processing entirely. Instead of processing tokens one at a time and maintaining a hidden state, the Transformer processes all tokens simultaneously and uses self-attention to let every token directly attend to every other token in a single operation. This achieves two things vanilla RNNs could not: (1) long-range dependencies are captured without gradient decay — token 1 and token 512 communicate directly via attention, not through 511 sequential hidden state updates; (2) the entire computation parallelises across the sequence dimension, enabling training on billion-parameter models that would be infeasible with sequential RNN computation. The Transformer encoder uses stacked self-attention blocks with residual connections and layer normalisation. The Transformer decoder adds causal (masked) self-attention and cross-attention to the encoder output. Understanding the Transformer architecture end-to-end — positional encoding, multi-head self-attention, feed-forward sublayers, residual connections, layer norm — is the prerequisite for understanding BERT, GPT, T5, and every large language model in production today.