What is the difference between mixed precision training and quantization?

Mixed precision training speeds up and reduces memory during model training by using lower-precision arithmetic while keeping an FP32 master weight copy, while quantization compresses an already-trained (or training) model's weights for efficient storage and inference.

Why is loss scaling needed in mixed precision training?

FP16's limited numerical range can cause small gradient values to underflow to zero; scaling the loss up before backpropagation and scaling gradients back down afterward keeps gradients within FP16's representable range.

What is the difference between FP16 and BF16?

FP16 has more mantissa bits (more precision) but a narrower exponent range prone to underflow, while BF16 has the same exponent range as FP32 (avoiding much underflow risk) but fewer mantissa bits, trading precision for range.

How much speedup does mixed precision training typically provide?

It commonly delivers 1.5x to 3x training speedups on hardware with dedicated low-precision compute units like NVIDIA tensor cores, alongside roughly halved memory usage.

Does mixed precision training hurt model accuracy?

In most cases, no; with proper loss scaling and framework support, mixed precision training achieves accuracy very close to full FP32 training while being significantly faster and more memory-efficient.

Which frameworks support mixed precision training out of the box?

PyTorch provides automatic mixed precision via torch.cuda.amp, and TensorFlow provides a similar mixed precision API, both of which automate loss scaling and precision selection.

AI Fundamentals

Mixed Precision Training

Training efficiency technique

IntermediateTechnique9.5K learners

Mixed precision training is a technique for training neural networks that uses lower-precision numerical formats (such as 16-bit floating point) for most computations while selectively retaining higher-precision 32-bit values where needed,…

#MixedPrecisionTraining #AIFundamentals#Technique#Intermediate#Quantization #ContrastiveLearning #MultiTaskLearning #FederatedLearning #ArtificialIntelligence#Glossary#SkillVeris

Definition

Overview

Mixed precision training combines FP16 (or BF16, bfloat16) and FP32 (32-bit floating point) arithmetic within a single training run: forward and backward passes are computed primarily in the lower-precision format, which uses half the memory of FP32 and runs substantially faster on hardware with dedicated low-precision compute units, such as NVIDIA GPU tensor cores, while a master copy of the model's weights is kept in FP32 to preserve numerical stability during the small, cumulative weight updates that occur during optimization. A key technical challenge with pure FP16 training is that its limited numerical range can cause small gradient values to underflow to zero, stalling learning. Mixed precision training addresses this with loss scaling: the loss value is multiplied by a scaling factor before backpropagation, which proportionally scales up the gradients into FP16's representable range, and then the gradients are scaled back down before the weight update is applied. Modern deep learning frameworks (PyTorch's `torch.cuda.amp`, TensorFlow's mixed precision API) automate this process, dynamically adjusting the loss scale during training and automatically selecting which operations run in reduced precision versus which remain in FP32 for stability (such as certain reduction and normalization operations). BF16, an alternative 16-bit format with the same exponent range as FP32 but fewer mantissa bits, avoids much of the underflow risk that motivates loss scaling in FP16, at the cost of somewhat lower precision per value, and has become popular for training very large models. Mixed precision training is now standard practice for training large-scale deep learning models, including large language models and vision transformers, because it can roughly halve memory usage and provide 1.5x to 3x training speedups on modern accelerator hardware with negligible accuracy degradation compared to full FP32 training. This efficiency gain directly enables training larger models, using larger batch sizes, or reducing training costs and carbon footprint, and it pairs naturally with other efficiency techniques like gradient checkpointing and distributed data-parallel training in large-scale training pipelines.

Key Concepts

Combines FP16/BF16 compute with an FP32 master copy of model weights
Roughly halves memory usage compared to full FP32 training
Delivers 1.5x-3x training speedups on tensor-core-equipped hardware
Uses loss scaling to prevent small gradients from underflowing in FP16
BF16 avoids much of FP16's underflow risk due to its wider exponent range
Automated in modern frameworks via APIs like PyTorch's torch.cuda.amp
Preserves accuracy close to full-precision training in most cases
Standard practice for training large language models and vision transformers

Use Cases

Training large language models and vision transformers at scale

Reducing GPU memory footprint to enable larger batch sizes

Speeding up training throughput on tensor-core-equipped GPUs

Lowering cloud compute costs and energy consumption for large training runs

Enabling training of larger models on the same hardware budget

Combining with gradient checkpointing and distributed training for efficiency

Frequently Asked Questions

From the Blog

AI & Technology

RAG Explained: How AI Answers From Your Data

RAG lets AI answer from your private documents instead of just its training data — here's how it works.

RAG Explained: Retrieval-Augmented Generation

RAG is how you give an LLM access to your own private data without training a new model. This guide explains the full pipeline — chunking, embeddings, vector search, and augmented generation — with a working Python example using open-source tools.

Fine-Tuning LLMs: A Practical Guide

Fine-tuning lets you adapt a pre-trained language model to your specific domain, style, or task — without training from scratch. This guide explains when fine-tuning is the right choice, how LoRA makes it affordable, and how to run a fine-tuning job with Hugging Face PEFT.

How Large Language Models Actually Work

LLMs seem magical until you understand what they are: next-token predictors trained on massive text corpora. This guide explains tokenisation, embeddings, the transformer architecture, attention mechanism, and how training works — without requiring a maths degree.