Gradient Accumulation
Gradient accumulation is a training technique that simulates a larger batch size than fits in available memory by summing gradients over several smaller mini-batches before performing a single optimizer update.
Definition
Gradient accumulation is a training technique that simulates a larger batch size than fits in available memory by summing gradients over several smaller mini-batches before performing a single optimizer update.
Overview
Larger batch sizes generally produce more stable gradient estimates and can improve training throughput, but they require proportionally more memory to hold activations for the forward and backward pass. GPU or TPU memory is often the binding constraint, especially when training large models like transformers, which forces practitioners to use smaller batch sizes than they would prefer. Gradient accumulation offers a workaround: instead of updating the model's weights after every mini-batch, the gradients from several consecutive mini-batches are accumulated (summed or averaged) and the optimizer step is only applied after a chosen number of accumulation steps. Mechanically, this means the forward and backward passes run normally for each small mini-batch, but the resulting gradients are added to a running total rather than immediately used to update weights, and the model's parameters are held fixed until enough mini-batches have been processed. Once the target number of accumulation steps is reached, the accumulated gradients are used for a single optimizer update, and the accumulator is reset to zero. The effective batch size becomes the mini-batch size multiplied by the number of accumulation steps, letting a team train with an effective batch size of, say, 512 while only ever holding activations for a mini-batch of 16 in memory at once. This is particularly valuable for large language model training and fine-tuning, where memory is the primary bottleneck and the desired batch size for stable, efficient training may be far larger than what a single GPU can hold. Gradient accumulation trades wall-clock training time for memory savings, since it requires multiple sequential forward/backward passes to achieve the update frequency of one large batch. It is commonly combined with mixed-precision training and memory-efficient optimizers to further reduce memory pressure, and most training frameworks (including Hugging Face Transformers and PyTorch Lightning) support it as a simple configuration flag.
Key Concepts
- Simulates a large effective batch size without needing the memory for it
- Sums or averages gradients over multiple mini-batches before an optimizer step
- Effective batch size equals mini-batch size times number of accumulation steps
- Trades additional wall-clock time for reduced memory usage
- Commonly paired with mixed-precision training to further save memory
- Widely supported via a simple configuration option in major training frameworks
- Especially useful for fine-tuning large language models on limited hardware
Use Cases
Frequently Asked Questions
From the Blog
What Is Gradient Descent Explained for Beginners
Gradient descent is the algorithm that trains most machine learning models by repeatedly nudging parameters in the direction that reduces error, step by step.
Read More Data ScienceA Practical Feature Engineering Playbook for Tabular Data
Feature engineering for tabular data is best organised by data type and model family, with a validation loop that proves each feature earns its place. Learn how to treat numeric, categorical, temporal and event data differently, avoid leakage, and decide which transformations gradient-boosted trees genuinely need versus which only linear models do.
Read More Data ScienceHow to encode high-cardinality categorical features without blowing up the model
Two axes decide the encoding: how many distinct values the column has, and which model family consumes it. Gradient-boosted trees often need no encoding at all, linear models need one-hot on a reduced vocabulary, and target encoding is safe only when computed out of fold.
Read More Data ScienceHow to fix CUDA out of memory in PyTorch without buying a bigger GPU
GPU memory splits into parameters, gradients, optimiser state and activations, and only the activation term responds to batch size. Measure the breakdown first, then apply remedies in that order: batch size and accumulation, gradient checkpointing, mixed precision, then a leaner optimiser.
Read More