# Quantization

> Storing a model's numbers with fewer bits so it needs less memory and runs faster.

- Level: Advanced
- Topics: AI, Hardware
- URL: https://tidbit.legios.com.ar/quantization/
- In Spanish: [Cuantización](https://tidbit.legios.com.ar/es/quantization/)

A trained model is billions of numbers, its weights, usually stored as 16-bit floating point (FP16 or BF16): two bytes each. **Quantization** maps them to a smaller format, typically 8-bit or 4-bit integers. A 70-billion-parameter model shrinks from roughly 140 GB of weights to roughly 35 GB at 4 bits: the difference between needing several data-center GPUs and fitting on one large one.

The core trick is a scale factor. Weights are split into small blocks, say 32 or 128 values; each block keeps one higher-precision scale, and each weight becomes a small integer that, multiplied by that scale, approximates the original. Rounding throws information away, so methods such as GPTQ and AWQ choose the rounding carefully using a little sample data and protect the weights that matter most. Formats like GGUF, used by llama.cpp and Ollama, package these schemes for running models locally.

Most quantization happens after training (**post-training quantization**) and takes minutes to hours. **Quantization-aware training** simulates the low precision during training so the model learns to cope with it, at a higher cost. Beyond weights, activations and the KV cache can be quantized too, and recent GPUs have hardware support for 8-bit and even 4-bit floating-point math.

The speedup comes mostly from memory, not arithmetic: generating each token means reading every weight, so moving a half or a quarter of the bytes makes generation faster before any math changes. The price is quality. 8 bits is usually close to lossless; 4 bits is a popular compromise with a small, measurable drop; below that, quality falls off quickly, and small models tend to suffer more than large ones. Evaluate a quantized model on your own task, not only on a public benchmark.

## An analogy

Saving a photo as a compressed JPEG instead of a RAW file. It takes a fraction of the space and looks almost the same; squeeze it too hard and blotches appear.

## Keep in mind

- Going from 16-bit to 4-bit cuts weight memory roughly by four.
- Block-wise scale factors keep the rounding error small.
- Faster generation comes mainly from reading fewer bytes per token.
- 8-bit is near lossless; below 4 bits quality drops fast.

## Go deeper

- [Quantization overview (Hugging Face Transformers docs)](https://huggingface.co/docs/transformers/main/en/quantization/overview)
- [Quantization concept guide (Hugging Face Optimum)](https://huggingface.co/docs/optimum/concept_guides/quantization)

## Related terms

- [Model parameters](https://tidbit.legios.com.ar/model-parameters/): The billions of numbers a model learns in training; '70B' means 70 billion of them.
- [Bit and byte](https://tidbit.legios.com.ar/bit-and-byte/): The smallest unit of digital information, and the group of eight that computers count in.
- [GPU](https://tidbit.legios.com.ar/gpu/): The chip that does thousands of simple calculations at once.
- [Training and inference](https://tidbit.legios.com.ar/training-and-inference/): Training builds the model once; inference is using it, millions of times a day.
- [Caching](https://tidbit.legios.com.ar/caching/): Keeping a copy of an answer close by so you do not have to work it out again.
- [Token](https://tidbit.legios.com.ar/token/): The small chunk of text a language model actually reads.