Batch Normalization
Batch normalization is a neural network training technique that normalizes the inputs to each layer using the mean and variance computed over a mini-batch, then rescales and shifts the result with learnable parameters.
Definition
Batch normalization is a neural network training technique that normalizes the inputs to each layer using the mean and variance computed over a mini-batch, then rescales and shifts the result with learnable parameters.
Overview
Introduced by Sergey Ioffe and Christian Szegedy in 2015, batch normalization addresses a problem the authors called "internal covariate shift" — the tendency for the distribution of each layer's inputs to change during training as the parameters of preceding layers update. By normalizing each mini-batch to zero mean and unit variance before applying two learnable parameters (a scale, gamma, and a shift, beta), the technique keeps activations in a well-behaved range throughout training. In practice, batch normalization layers are inserted between a linear or convolutional layer and its activation function. During training, statistics are computed per mini-batch; at inference time, the layer instead uses running averages of the mean and variance accumulated during training, so predictions do not depend on batch composition. This distinction between train-mode and eval-mode behavior is a common source of bugs when models are deployed incorrectly. The main practical benefits are faster convergence, tolerance for higher learning rates, and a mild regularizing effect from the noise introduced by mini-batch statistics, which can reduce (though not eliminate) the need for other regularization such as dropout. Its main drawback is a dependency on batch size — performance degrades with very small batches, which is why alternatives like layer normalization, group normalization, and instance normalization were developed for settings such as recurrent networks, transformers, and small-batch or per-example inference. Batch normalization remains a default building block in convolutional architectures such as ResNet, while transformer-based models overwhelmingly favor layer normalization instead.
Key Concepts
- Normalizes layer inputs using per-mini-batch mean and variance
- Learnable scale (gamma) and shift (beta) parameters restore representational flexibility after normalization
- Uses running statistics accumulated during training for inference-time normalization
- Allows higher learning rates and generally speeds up convergence
- Provides a mild regularization effect from mini-batch noise
- Placed between a linear/convolutional layer and its activation function
- Performance depends on batch size; degrades with very small batches
- Standard component in CNN architectures like ResNet and VGG variants
Use Cases
Frequently Asked Questions
From the Blog
What Is an Epoch, Batch and Iteration in Training
An epoch is one full pass over your training data, a batch is a slice of it, and an iteration is one weight update. Here is how the three fit together.
Read More Data ScienceData Normalization vs Standardization Explained
Normalization scales data to a fixed range; standardization rescales to zero mean and unit variance. Learn when to use each and how to avoid data leakage.
Read More AI & TechnologyHow to Re-Embed a Corpus When You Change Models
Re-embed by writing the new vectors into a separate versioned collection, backfilling in batches while the old collection continues serving, then cutting over behind a config flag. This article covers the mixing hazard, batch and checkpoint design, dual-write during backfill, and how to validate before you switch.
Read More AI & TechnologyContinuous batching: raising LLM throughput without hurting latency
Continuous batching refills the batch every decoding step. Learn why it beats static batching, how queueing sets tail latency, and which knobs to tune first.
Read More