Quantization means representing the numbers inside a model with fewer bits. A trained model's weights are usually 16-bit floats; quantization compresses them to 8-bit, 4-bit, or sometimes lower integers. Fewer bits means a smaller file, less memory used, faster movement of data between memory and compute units — and therefore faster inference.
It's the first concept anyone running models locally runs into. The Q4_K_M, INT8, AWQ, and GPTQ labels on model download pages all describe a quantization scheme and bit width. The same model that needs 140GB of memory at full precision may fit in a little over 40GB at 4 bits — often the difference between running on your own machine and not.
Grab It in One Sentence First
Quantization trims decimal places off a model's weights, trading a little precision for memory and speed.
The everyday analogy is photo compression. A RAW file is tens of megabytes; saved as JPEG it's a few, and to the eye it's nearly identical — but zoom to the pixel level and detail has been merged. Quantization does the same: in most cases the output is indistinguishable, but where fine judgment matters most, the loss shows.
What It Actually Saves
A 70B-parameter model stored as 16-bit floats needs roughly 140GB just for weights. The arithmetic is simple:
| Precision | Bytes per parameter | Weights for a 70B model |
|---|---|---|
| FP16 / BF16 (16-bit) | 2 bytes | ~140 GB |
| INT8 (8-bit) | 1 byte | ~70 GB |
| INT4 (4-bit) | 0.5 bytes | ~35 GB |
Real usage adds KV cache, activations, and framework overhead, so requirements run higher. But the trend is clear: halve the bit width, roughly halve the memory.
The source of the speedup is often misunderstood. During generation, large-model inference is mostly memory-bandwidth bound — the bottleneck isn't how fast you can compute but how fast weights move from memory into compute units. Smaller weights mean less to move, so generation speeds up. That's also why quantization improves per-token generation speed more visibly than it improves time-to-first-token.
The Common Approaches
Post-training quantization (PTQ) is what most people encounter: the model is already trained, a small calibration set determines the scaling factors, and the weights are compressed directly. GPTQ and AWQ are two common implementations, and the GGUF files in the llama.cpp ecosystem belong to this family. It's fast and requires no retraining.
Quantization-aware training (QAT) simulates low-precision error during training so the model adapts to it. Results are usually better, but it costs training resources, so generally only model publishers do it.
Mixed precision is the norm in practice: not all layers are equally sensitive, so certain attention projections plus the embedding and output layers often keep higher precision while the rest drops to 4 bits. The M (medium) in a GGUF name like Q4_K_M describes exactly this kind of mixed strategy.
How Much Quality Is Lost
A few reasonably stable rules of thumb, all of which still need verifying on your own task:
- 8-bit is usually near-lossless and safe to adopt.
- 4-bit loses very little on most general tasks and is the mainstream choice for local deployment.
- Below 4 bits (3-bit, 2-bit) the loss becomes noticeable; generally reserved for severe memory pressure.
- Bigger models tolerate quantization better. At the same 4-bit target, a 70B model typically degrades less in relative terms than a 7B one. So "a large model at 4 bits" often beats "a small model at full precision" for the same memory budget.
An important caveat: the loss isn't evenly distributed. A quantized model indistinguishable in casual Q&A may degrade sharply on long reasoning chains, exact numerical work, code generation, and lower-resource languages. Those differences have to be measured with your own evals, not inferred from someone else's perplexity numbers.
Versus Distillation, Small Models, and Fine-Tuning
Versus distillation. Distillation trains a genuinely smaller new model to imitate a large one, so parameter count really drops. Quantization changes neither parameter count nor architecture, only the precision of each parameter. The two can be stacked.
Versus small language models. An SLM is small by design; quantization shrinks an existing large model. At the same memory budget they are genuine competitors, and the choice should be made by measurement on your task.
Its relationship to LoRA. QLoRA combines the two — freeze a base model quantized to 4 bits and train only a small set of LoRA parameters, letting consumer GPUs fine-tune large models.
Where People Get It Wrong
"Quantization makes models dumber." Too broad. 8-bit is essentially lossless and 4-bit is usable in most settings. The trouble comes from chasing ever-lower bit widths, or applying aggressive quantization to precision-sensitive tasks without testing.
"Quantization only matters for local deployment." Cloud services use quantization and low-precision inference extensively to control cost — it's just invisible to users. When a model with the same name behaves inconsistently on an API relay or an opaque third-party service, a quantization difference is a common cause.
"Smaller file, therefore faster." If the model still doesn't fit in GPU memory and has to page between system and device memory, throughput collapses. The speedup assumes the whole model plus KV cache fits.
When to Use It
To run a local model on your own machine, quantization is effectively mandatory; start at 4 bits. For cloud services it's a routine cost lever, but compare against the full-precision version on real tasks before launch, focusing on the hardest samples in your workload.
If you're evaluating a third-party API, add "what precision do you serve?" to your due-diligence list. It rarely appears on a pricing page and directly affects the quality you receive.