Mixed Precision Training
Mixed precision training is a technique for training neural networks that uses lower-precision numerical formats (such as 16-bit floating point) for most computations while selectively retaining higher-precision 32-bit values where needed, to speed up training and reduce memory usage without sacrificing model accuracy.
3 resources across 1 library
Glossary Terms(3)
Quantization (ML)
Quantization is a model compression technique that reduces the numerical precision used to represent a neural network's weights and activations (e.g., from 32-…
Mixed Precision Training
Mixed precision training is a technique for training neural networks that uses lower-precision numerical formats (such as 16-bit floating point) for most compu…
Gradient Accumulation
Gradient accumulation is a training technique that simulates a larger batch size than fits in available memory by summing gradients over several smaller mini-b…