Layer Normalization
Layer normalization is a neural network normalization technique that normalizes activations across the features of a single training example, rather than across a mini-batch, making it independent of batch size.
Definition
Layer normalization is a neural network normalization technique that normalizes activations across the features of a single training example, rather than across a mini-batch, making it independent of batch size.
Overview
Proposed by Jimmy Ba, Jamie Ryan Kiros, and Geoffrey Hinton in 2016, layer normalization was designed to address a key limitation of batch normalization: its dependence on mini-batch statistics, which makes it awkward for recurrent networks, small-batch training, and inference on a single example. Instead of normalizing across the batch dimension, layer normalization computes the mean and variance across all the features within one training example (e.g., across the hidden units of a single token's representation) and normalizes using those statistics. Because the normalization statistics are computed per example rather than per batch, layer normalization behaves identically during training and inference and works with a batch size of one. This property made it the default normalization choice in transformer architectures, including the original "Attention Is All You Need" model, BERT, GPT, and virtually all subsequent large language models. Like batch normalization, it uses learnable scale and shift parameters applied after normalization. A notable architectural detail is where layer normalization is placed relative to the sub-layers of a transformer block. The original transformer used "post-norm," applying normalization after the residual addition; most modern large language models use "pre-norm," applying it before each sub-layer, which tends to produce more stable gradients in very deep stacks. Variants such as RMSNorm, used in models like LLaMA, simplify the computation by normalizing only using the root-mean-square of activations and dropping the mean-centering step, offering a small efficiency gain with comparable performance.
Key Concepts
- Normalizes across the feature dimension of a single example, not across a batch
- Identical behavior during training and inference, unlike batch normalization
- Works correctly with a batch size of one, including autoregressive inference
- Learnable per-feature scale and shift parameters after normalization
- Standard component in transformer architectures (BERT, GPT, and successors)
- Supports pre-norm and post-norm placement relative to residual connections
- RMSNorm is a widely used simplified variant that drops mean-centering
Use Cases
Frequently Asked Questions
From the Blog
Vector Databases Explained: The Memory Layer Powering AI Apps
Vector databases are the storage layer behind RAG systems, semantic search, and AI- powered recommendations. This guide explains what they are, how they differ from traditional databases, and how to choose and use one in a real application.
Read More Data ScienceData Normalization vs Standardization Explained
Normalization scales data to a fixed range; standardization rescales to zero mean and unit variance. Learn when to use each and how to avoid data leakage.
Read More AI & TechnologyUI Design Explained: Principles and Practice
User interface design is the practice of designing the visual and interactive layer of a product so it is clear, consistent, and usable. This guide explains core UI principles, how UI differs from UX, and how to start designing interfaces.
Read More Cloud & CybersecurityThe Network Layer Explained: How Data Finds Its Way
The network layer is the part of a computer network that decides how data packets travel from a source to a destination across interconnected networks. This guide explains its job, the protocols it relies on, and why routing and addressing sit at its core.
Read More