Transformer Architecture
The Transformer is a neural network architecture built entirely around the self-attention mechanism, allowing it to model relationships between all elements of an input sequence in parallel rather than processing them one at a time.
Definition
The Transformer is a neural network architecture built entirely around the self-attention mechanism, allowing it to model relationships between all elements of an input sequence in parallel rather than processing them one at a time.
Overview
Introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al., the Transformer replaced the recurrent and convolutional architectures that previously dominated sequence modeling, such as LSTMs and RNNs, which processed tokens sequentially and struggled to capture long-range dependencies efficiently. Instead, the Transformer relies on self-attention, a mechanism that lets every token in a sequence directly attend to every other token, computing a weighted representation based on relevance regardless of distance in the sequence. This parallelizability made Transformers dramatically more efficient to train on modern hardware like GPUs and TPUs, which was a key enabler of the scaling that later produced large language models. The original architecture consists of an encoder stack and a decoder stack, each built from repeated layers combining multi-head self-attention (which lets the model attend to different types of relationships simultaneously through parallel attention "heads") with position-wise feedforward networks, residual connections, and layer normalization. Because self-attention has no inherent sense of token order, positional encodings are added to the input embeddings to inject sequence position information. Many modern LLMs use only the decoder portion (decoder-only architectures like GPT), or only the encoder portion (encoder-only architectures like BERT), depending on whether the task is generative or primarily about understanding input. The Transformer's efficiency and expressiveness made it the dominant architecture across not just NLP but also computer vision (via Vision Transformers), speech, and multimodal models, effectively replacing task-specific architectures across much of deep learning. Its main computational limitation is that standard self-attention scales quadratically with sequence length, since every token must attend to every other token, which has motivated a large body of research into more efficient attention variants — such as sparse attention, linear attention, and sliding window attention — to support longer context windows without prohibitive compute costs.
Key Concepts
- Built entirely around the self-attention mechanism, without recurrence
- Processes all tokens in a sequence in parallel rather than sequentially
- Multi-head attention captures multiple types of relationships simultaneously
- Uses positional encodings to inject sequence order information
- Composed of encoder and/or decoder stacks with residual connections and layer normalization
- Introduced in "Attention Is All You Need" (Vaswani et al., 2017)
- Standard self-attention scales quadratically with sequence length
- Foundation for GPT, BERT, Vision Transformers, and most modern LLMs
Use Cases
Frequently Asked Questions
From the Blog
Inside the Transformer: Every Block, Explained in Order
Follow a single token through a transformer: tokenisation, embedding, positional information, attention, the feed-forward block, normalisation, and the final projection to a probability distribution. By the end you will know what every field in a model config controls and where training instability comes from.
Read More AI & TechnologyHow Large Language Models Actually Work
LLMs seem magical until you understand what they are: next-token predictors trained on massive text corpora. This guide explains tokenisation, embeddings, the transformer architecture, attention mechanism, and how training works — without requiring a maths degree.
Read More AI & TechnologyWhat Is Self-Attention in Neural Networks
Self-attention lets each token in a sequence attend to every other token in the same sequence, building context-aware representations that power transformer models.
Read More AI & TechnologyWhat Are Positional Encodings in Transformers
Positional encodings tell a transformer the order of its tokens, since self-attention alone is order-blind. Learn how sinusoidal, learned, and rotary variants work.
Read More