What Is Model Quantization and Why It Matters
SkillVeris Team
AI Research Team
Model quantization stores a neural network's weights in lower-precision numbers, shrinking its memory footprint and speeding up inference with usually minor accuracy loss.
In this guide, you'll learn:
- A typical example is converting 16-bit weights down to 8-bit or 4-bit integers, cutting model size by half or more.
- Smaller models fit on cheaper hardware, load faster, and run on phones and laptops that could not hold the full-precision version.
- Post-training quantization is applied after training and is fast; quantization-aware training bakes it in during training for better accuracy.
- The trade-off is a small drop in quality that grows as you push to very low bit widths.
1What Is Model Quantization?
Model quantization is the process of representing a neural network's numbers — mainly its weights — using fewer bits of precision, which shrinks the model and speeds up inference. Instead of storing each weight as a 32-bit or 16-bit floating-point value, you store it as an 8-bit or even 4-bit integer, trading a little accuracy for large gains in size and speed.
The idea works because neural networks are surprisingly tolerant of imprecision. The exact value of any single weight rarely matters much, so rounding them to a coarser set of levels usually changes the output only slightly while dramatically reducing memory and compute.
2Why It Matters
Quantization matters because model size directly determines what hardware can run a model and how fast. A large model in full precision may need expensive accelerators with lots of memory, while its quantized version can fit on a single consumer GPU, a laptop, or even a phone.
- Smaller memory footprint: a 4-bit model can be a quarter the size of a 16-bit one.
- Faster inference: less data to move means quicker responses and higher throughput.
- Cheaper hardware: fits on commodity GPUs or CPUs instead of top-tier accelerators.
- On-device AI: enables models to run locally on phones and laptops, offline and private.
🔑Key Point
Quantization is a big reason capable open models now run on a laptop. Halving or quartering size turns a data-centre model into a local one.
3How Precision Works
Numbers in a model are stored in formats with different bit widths, and fewer bits means fewer distinct values a weight can take. Quantization maps the wide range of a high-precision format onto a smaller set of integer levels, using a scale factor to translate back and forth.
- FP32: 32-bit float — full precision, largest, used in training.
- FP16 / BF16: 16-bit float — common inference default, half the size.
- INT8: 8-bit integer — roughly a quarter of FP32, small accuracy cost.
- INT4: 4-bit integer — very compact, larger but often acceptable accuracy cost.
The Scale Factor
Because integers cannot represent the full floating-point range directly, quantization stores a scale (and sometimes an offset) per group of weights. At inference the integer is multiplied back by the scale to approximate the original value.
4Post-Training vs Quantization-Aware Training
There are two main ways to quantize, differing in when the quantization happens and how much accuracy they preserve.
💡Start With PTQ
Try post-training quantization first — it is fast and often good enough. Reach for quantization-aware training only if accuracy at your target bit width falls short.
Post-Training Quantization (PTQ)
PTQ takes a fully trained model and converts its weights to lower precision afterward, often in minutes. It needs no retraining and little data, which makes it the common choice, though very low bit widths can lose more accuracy.
Quantization-Aware Training (QAT)
QAT simulates low-precision arithmetic during training so the model learns to compensate. It is more work and needs the training pipeline, but it typically recovers more accuracy, especially at aggressive bit widths.
5The Accuracy Trade-off
Quantization is not free — pushing to fewer bits eventually degrades quality, and the loss grows as precision drops. Eight-bit quantization is usually nearly lossless for many models, while four-bit and below can show noticeable degradation depending on the model and method.
Modern techniques soften this by quantizing different parts of the model to different precisions, keeping sensitive layers higher and squeezing tolerant ones harder. Always evaluate the quantized model on real tasks rather than assuming the drop is negligible.
6Common Mistakes to Avoid
Quantization is approachable, but a few missteps cause avoidable quality loss.
- Pushing straight to 4-bit without checking whether 8-bit already meets your needs.
- Skipping evaluation and assuming the quantized model performs like the original.
- Quantizing every layer uniformly when some are far more sensitive than others.
- Ignoring hardware support — some formats only accelerate on specific chips.
- Confusing quantization with pruning or distillation; they are different compression methods.
⚠️Watch Out
Accuracy loss is model- and task-dependent. A quantization that is harmless for chat may hurt a precise extraction task — always test on your workload.
7Quantization vs Other Compression
Quantization is one of several ways to shrink a model, and knowing how it differs from the alternatives helps you combine them well. Pruning removes weights or whole neurons that contribute little, while distillation trains a smaller student model to imitate a larger teacher. Each attacks size from a different angle.
- Quantization: fewer bits per weight — the fastest, most common first step.
- Pruning: remove low-importance weights or structures entirely.
- Distillation: train a smaller model to mimic a larger one's outputs.
- These can be stacked — a distilled model can also be quantized for further gains.
8Key Takeaways
Quantization is a core tool for making models cheaper and more portable.
- It stores weights in lower precision to shrink models and speed inference.
- Going from 16-bit to 8-bit or 4-bit can cut size by half or more.
- Smaller models run on cheaper hardware and on-device, offline and private.
- PTQ is fast and applied after training; QAT preserves more accuracy during training.
- Very low bit widths trade quality for size, so always evaluate on your real task.
9Frequently Asked Questions
Q: Does quantization make a model less accurate? A: It usually causes a small drop that grows as you use fewer bits. Eight-bit quantization is often nearly lossless, while four-bit and below can show noticeable degradation depending on the model, method, and task.
Q: What is the difference between PTQ and QAT? A: Post-training quantization converts an already trained model to lower precision quickly with little data. Quantization-aware training simulates low precision during training so the model adapts, generally preserving more accuracy at aggressive bit widths.
Q: Why does quantization speed up inference? A: Lower-precision numbers are smaller and faster to move and compute, so there is less memory traffic and, on supporting hardware, faster arithmetic. The main win is often reduced memory bandwidth.
Q: Is quantization the same as pruning? A: No. Quantization reduces the precision of numbers, while pruning removes weights or neurons entirely. Both compress models but through different mechanisms, and they can be combined.
Related Reading
Get The Print Version
Download a PDF of this article for offline reading.
About the Publisher
SkillVeris Team
AI Research Team
Our AI team covers the latest in machine learning, generative AI, and emerging tech — clearly and accurately.
View all postsRelated Posts
Never miss an update
Get the latest tutorials and guides delivered to your inbox.
No spam. Unsubscribe anytime.