Batch Normalization Cheat Sheet

Explains why batch normalization stabilizes training, its learnable scale and shift parameters, and how train versus eval mode statistics differ in PyTorch.

1 PageIntermediateMar 10, 2026

Core Concepts

What batch norm computes and why.

Normalization step- For each mini-batch, subtract the batch mean and divide by the batch standard deviation, per feature/channel
Learnable scale/shift (gamma, beta)- After normalizing, the layer applies y = gamma*x_hat + beta so it can undo normalization if that's optimal
Internal covariate shift- The original motivation: reduce how much the distribution of layer inputs shifts as earlier layers update
Running statistics- During training, exponential moving averages of batch mean/variance are tracked for use at inference
Train vs. eval mode- Training uses current batch statistics; eval (model.eval()) uses the stored running mean/variance instead

Batch Normalization in PyTorch

Typical placement in a CNN block.

python

import torch.nn as nnblock = nn.Sequential(    nn.Conv2d(64, 128, kernel_size=3, padding=1),    nn.BatchNorm2d(128),   # one gamma/beta per channel    nn.ReLU(inplace=True),)# Critical: switch modes correctlymodel.train()   # uses batch statistics, updates running statsmodel.eval()    # uses stored running_mean / running_var, no updates

The Batch Norm Formula

What happens under the hood for a single feature.

python

# x: activations for one feature across the mini-batchmu = x.mean()var = x.var(unbiased=False)x_hat = (x - mu) / (var + eps).sqrt()   # eps avoids divide-by-zero, e.g. 1e-5y = gamma * x_hat + beta                # gamma, beta are learned per-feature

Practical Notes

Common pitfalls.

Small batch sizes- Batch norm statistics get noisy with very small batches (e.g., < 8); consider GroupNorm or LayerNorm instead
Forgetting model.eval()- The single most common inference bug -- leaves the model using batch statistics from the (often batch-size-1) inference input
Placement relative to activation- Commonly Conv/Linear -> BatchNorm -> Activation, though BN-after-activation is also used in some architectures
Bias term redundancy- The preceding Conv/Linear layer's bias is usually disabled (bias=False) since BatchNorm's beta already shifts the output

Pro Tip

Batch norm's running statistics are only updated during model.train() forward passes -- if you forget to call model.train() before a training loop, you'll silently train against stale statistics.

Was this cheat sheet helpful?

Deep Learning Optimizers Cheat Sheet

Transfer Learning Cheat Sheet

Explore Topics

#BatchNormalization#BatchNormalizationCheatSheet#DataScience#Intermediate#CoreConcepts#BatchNormalizationInPyTorch#TheBatchNormFormula#PracticalNotes#MachineLearning#CheatSheet#SkillVeris

Sri Hayavadhana Info-Tech

Professional Web Designing Services