Quantization Explained: Shrinking Models Without Losing Power

1What Is Quantization?

Quantization is the process of converting numbers stored in a high-precision format, such as 32-bit floating point, into a lower-precision format, such as 8-bit integers, while trying to preserve the information those numbers represent. In machine learning, it is applied to a model's weights and sometimes its activations so the model takes up less memory and runs faster.

The core idea is that neural networks are often tolerant of small numerical errors, so replacing exact floating-point values with a compact approximation rarely changes the model's output in a meaningful way, but it can dramatically reduce the compute and memory the model needs.

2The Quantization Parameter Explained

The quantization parameter is the pair of values, scale and zero-point, that defines how a range of floating-point numbers maps onto a smaller range of integers. The scale controls how much each integer step represents in the original value space, and the zero-point marks which integer corresponds to a real value of zero.

Once scale and zero-point are fixed for a tensor, converting between the original float value and its quantized integer becomes a simple, fast arithmetic operation that hardware can execute far more efficiently than full floating-point math.

• Scale: determines the step size between adjacent quantized values.
• Zero-point: the integer value that represents a real number of exactly zero.
• Bit width: how many bits are used per value, commonly 8, 4, or even fewer for aggressive compression.
• Symmetric vs asymmetric: symmetric quantization centers the range around zero; asymmetric shifts it to better fit skewed data distributions.

3Why Quantization Matters

Quantization matters because it directly controls how much memory a model consumes and how fast it can run, which decides whether a model can realistically be deployed on a given device at all.

• Smaller footprint: an 8-bit model can be roughly a quarter the size of its 32-bit counterpart.
• Faster inference: integer arithmetic is generally cheaper for hardware to execute than floating-point arithmetic.
• Lower power draw: fewer bits moved and processed means less energy used, which matters for mobile and edge devices.
• Higher throughput: smaller models allow more requests to be served per unit of compute in production.

4Post-Training Quantization vs Quantization-Aware Training

There are two broad approaches to quantizing a model, and the choice between them comes down to how much accuracy you can afford to trade for convenience.

Post-training quantization takes an already-trained model and converts its weights afterward, using a small calibration dataset to pick good scale and zero-point values. It is fast and requires no retraining, but for some models it introduces a noticeable accuracy drop.

5Per-Tensor vs Per-Channel Quantization

Not every value in a model needs to share the same scale and zero-point. How granularly you apply the quantization parameter affects both accuracy and computational overhead.

• Per-tensor quantization: one scale and zero-point for an entire weight tensor, simplest and fastest.
• Per-channel quantization: a separate scale and zero-point for each output channel, which better handles weights whose ranges vary a lot across channels.
• Per-channel typically costs a little more bookkeeping but recovers accuracy that per-tensor quantization would otherwise lose.

6Quantizing Weights and Activations

Weights and activations can be quantized independently, and doing both is usually needed to get the full speed benefit, since integer hardware paths only kick in when both operands of a computation are integers.

Activations are trickier than weights because their range depends on the input data at runtime, which is why calibration passes over representative data are used to estimate a stable range before deployment.

7Common Precision Levels in Practice

Different bit widths suit different deployment constraints, and picking one is a direct trade-off between size, speed, and accuracy.

• FP16 / BF16: half the size of FP32 with minimal accuracy loss, common for training and GPU inference.
• INT8: a widely supported sweet spot offering strong speedups with modest, often negligible, accuracy loss.
• INT4 and below: aggressive compression used for very large models, where accuracy loss becomes more noticeable and needs careful evaluation.
• Mixed precision: keeping sensitive layers in higher precision while quantizing the rest more aggressively.

8Where Quantization Shows Up in Real Systems

Quantization is not a niche technique; it underlies most of the AI running on everyday devices. Understanding it connects directly to the broader computing fundamentals of how numbers are represented and processed in hardware.

Voice assistants, on-device photo classification, and local language model inference all depend on quantized models to run within the memory and battery limits of a phone or laptop.

9Getting Started with Quantization

Start by measuring your model's current size, latency, and accuracy so you have a baseline to compare against. Apply post-training quantization first since it is the least effort, and only move to quantization-aware training if the accuracy drop is unacceptable for your use case.

From there, experiment with per-channel quantization and mixed precision for the layers that are most sensitive, and always validate on real evaluation data before shipping a quantized model to production.