Quantization

A trained model is billions of numbers, its weights, usually stored as 16-bit floating point (FP16 or BF16): two bytes each. Quantization maps them to a smaller format, typically 8-bit or 4-bit integers. A 70-billion-parameter model shrinks from roughly 140 GB of weights to roughly 35 GB at 4 bits: the difference between needing several data-center GPUs and fitting on one large one.

The core trick is a scale factor. Weights are split into small blocks, say 32 or 128 values; each block keeps one higher-precision scale, and each weight becomes a small integer that, multiplied by that scale, approximates the original. Rounding throws information away, so methods such as GPTQ and AWQ choose the rounding carefully using a little sample data and protect the weights that matter most. Formats like GGUF, used by llama.cpp and Ollama, package these schemes for running models locally.

Most quantization happens after training (post-training quantization) and takes minutes to hours. Quantization-aware training simulates the low precision during training so the model learns to cope with it, at a higher cost. Beyond weights, activations and the KV cache can be quantized too, and recent GPUs have hardware support for 8-bit and even 4-bit floating-point math.

The speedup comes mostly from memory, not arithmetic: generating each token means reading every weight, so moving a half or a quarter of the bytes makes generation faster before any math changes. The price is quality. 8 bits is usually close to lossless; 4 bits is a popular compromise with a small, measurable drop; below that, quality falls off quickly, and small models tend to suffer more than large ones. Evaluate a quantized model on your own task, not only on a public benchmark.