Layer Normalization
Layer normalization is a neural network normalization technique that normalizes activations across the features of a single training example, rather than across a mini-batch, making it independent of batch size.
Definition
Layer normalization is a neural network normalization technique that normalizes activations across the features of a single training example, rather than across a mini-batch, making it independent of batch size.
Overview
Proposed by Jimmy Ba, Jamie Ryan Kiros, and Geoffrey Hinton in 2016, layer normalization was designed to address a key limitation of batch normalization: its dependence on mini-batch statistics, which makes it awkward for recurrent networks, small-batch training, and inference on a single example. Instead of normalizing across the batch dimension, layer normalization computes the mean and variance across all the features within one training example (e.g., across the hidden units of a single token's representation) and normalizes using those statistics. Because the normalization statistics are computed per example rather than per batch, layer normalization behaves identically during training and inference and works with a batch size of one. This property made it the default normalization choice in transformer architectures, including the original "Attention Is All You Need" model, BERT, GPT, and virtually all subsequent large language models. Like batch normalization, it uses learnable scale and shift parameters applied after normalization. A notable architectural detail is where layer normalization is placed relative to the sub-layers of a transformer block. The original transformer used "post-norm," applying normalization after the residual addition; most modern large language models use "pre-norm," applying it before each sub-layer, which tends to produce more stable gradients in very deep stacks. Variants such as RMSNorm, used in models like LLaMA, simplify the computation by normalizing only using the root-mean-square of activations and dropping the mean-centering step, offering a small efficiency gain with comparable performance.
Key Concepts
- Normalizes across the feature dimension of a single example, not across a batch
- Identical behavior during training and inference, unlike batch normalization
- Works correctly with a batch size of one, including autoregressive inference
- Learnable per-feature scale and shift parameters after normalization
- Standard component in transformer architectures (BERT, GPT, and successors)
- Supports pre-norm and post-norm placement relative to residual connections
- RMSNorm is a widely used simplified variant that drops mean-centering