Mixed Precision Training
Training efficiency technique
Mixed precision training is a technique for training neural networks that uses lower-precision numerical formats (such as 16-bit floating point) for most computations while selectively retaining higher-precision 32-bit values where needed,…
Definition
Mixed precision training is a technique for training neural networks that uses lower-precision numerical formats (such as 16-bit floating point) for most computations while selectively retaining higher-precision 32-bit values where needed, to speed up training and reduce memory usage without sacrificing model accuracy.
Overview
Mixed precision training combines FP16 (or BF16, bfloat16) and FP32 (32-bit floating point) arithmetic within a single training run: forward and backward passes are computed primarily in the lower-precision format, which uses half the memory of FP32 and runs substantially faster on hardware with dedicated low-precision compute units, such as NVIDIA GPU tensor cores, while a master copy of the model's weights is kept in FP32 to preserve numerical stability during the small, cumulative weight updates that occur during optimization. A key technical challenge with pure FP16 training is that its limited numerical range can cause small gradient values to underflow to zero, stalling learning. Mixed precision training addresses this with loss scaling: the loss value is multiplied by a scaling factor before backpropagation, which proportionally scales up the gradients into FP16's representable range, and then the gradients are scaled back down before the weight update is applied. Modern deep learning frameworks (PyTorch's `torch.cuda.amp`, TensorFlow's mixed precision API) automate this process, dynamically adjusting the loss scale during training and automatically selecting which operations run in reduced precision versus which remain in FP32 for stability (such as certain reduction and normalization operations). BF16, an alternative 16-bit format with the same exponent range as FP32 but fewer mantissa bits, avoids much of the underflow risk that motivates loss scaling in FP16, at the cost of somewhat lower precision per value, and has become popular for training very large models. Mixed precision training is now standard practice for training large-scale deep learning models, including large language models and vision transformers, because it can roughly halve memory usage and provide 1.5x to 3x training speedups on modern accelerator hardware with negligible accuracy degradation compared to full FP32 training. This efficiency gain directly enables training larger models, using larger batch sizes, or reducing training costs and carbon footprint, and it pairs naturally with other efficiency techniques like gradient checkpointing and distributed data-parallel training in large-scale training pipelines.
Key Concepts
- Combines FP16/BF16 compute with an FP32 master copy of model weights
- Roughly halves memory usage compared to full FP32 training
- Delivers 1.5x-3x training speedups on tensor-core-equipped hardware
- Uses loss scaling to prevent small gradients from underflowing in FP16
- BF16 avoids much of FP16's underflow risk due to its wider exponent range
- Automated in modern frameworks via APIs like PyTorch's torch.cuda.amp
- Preserves accuracy close to full-precision training in most cases
- Standard practice for training large language models and vision transformers
Use Cases
Frequently Asked Questions
From the Blog
What Is Precision and Recall Explained Simply
Precision measures how many of your model's positive predictions were correct; recall measures how many actual positives it managed to catch. Here's the simple version.
Read More AI & TechnologyWhat Is an Epoch, Batch and Iteration in Training
An epoch is one full pass over your training data, a batch is a slice of it, and an iteration is one weight update. Here is how the three fit together.
Read More AI & TechnologyCompetency-Based Training: How It Works and Why It Sticks
Competency-based training measures progress by demonstrated skill mastery instead of time spent in a classroom. This guide explains how it works, how it differs from traditional training, and how to implement it effectively.
Read More AI & TechnologyPersonalized Learning: How Tailored Training Works
Personalized learning adapts what, when, and how someone studies based on their existing knowledge, pace, and goals instead of a fixed curriculum. This guide explains how it works, what powers it, and how to evaluate it in a course.
Read More