Model Quantization

Definition

Model quantization is a technique that reduces the numeric precision used to represent a model's weights and activations, shrinking its size and speeding up inference with minimal accuracy loss.

Overview

Neural network weights are typically stored and computed using 32-bit floating-point numbers during training. Quantization converts these values to lower-precision formats — such as 16-bit, 8-bit, or even 4-bit integers — which reduces the memory footprint of the model and allows computations to run faster on compatible hardware, since lower-precision arithmetic requires less memory bandwidth and can be processed more efficiently. The main challenge is preserving accuracy: reducing precision too aggressively can degrade a model's outputs. Techniques like post-training quantization, which quantizes an already-trained model, and quantization-aware training, which simulates lower precision during training so the model adapts to it, are both used to minimize this tradeoff. In practice, many models can be quantized to 8-bit or even 4-bit precision with only a small, often acceptable, loss in quality. Quantization is one of the primary techniques enabling large models to run efficiently on consumer hardware, mobile devices, and edge deployments where memory and compute are limited. It is frequently combined with model distillation and model pruning as part of a broader model-compression strategy for deploying foundation models outside of large data-center environments.

Key Concepts

• Reduces numeric precision of model weights and activations
• Shrinks model size and speeds up inference on compatible hardware
• Post-training quantization applies to an already-trained model
• Quantization-aware training adapts the model to lower precision during training
• Enables running large models on consumer and edge hardware
• Often combined with distillation and pruning for further compression

Use Cases

Frequently Asked Questions

Related in AI Fundamentals