# Model distillation

> Training a small, cheap model to imitate a big, expensive one.

- Level: Advanced
- Topics: AI
- URL: https://tidbit.legios.com.ar/model-distillation/
- In Spanish: [Destilación de modelos](https://tidbit.legios.com.ar/es/model-distillation/)

**Model distillation**, or knowledge distillation, is a way to squeeze much of what a large model knows into a smaller one. You take a big, capable **teacher model**, run it on lots of inputs, and train a small **student model** to reproduce the teacher's outputs instead of learning only from the original labeled data.

The trick, described in a 2015 paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, is that the teacher's full probabilities carry more information than a single right answer. A photo classifier that says "cat 90%, fox 8%, car 0.01%" is also telling the student which mistakes are reasonable. With today's LLMs the same idea often takes a simpler form: the teacher generates many answers, or full step-by-step reasoning, and that text becomes training data for the student. In early 2025, for example, DeepSeek published several small open models distilled from its R1 reasoning model.

The payoff is at inference time. A distilled model has far fewer parameters, so it answers faster, costs less per request and can run on a laptop or a phone. The limits: the student rarely surpasses its teacher, it tends to lose the most when tasks drift away from what it was distilled on, and many commercial providers forbid in their terms using their models' outputs to train competing models.

## An analogy

A master chef who cannot open a restaurant in every city, so she trains junior cooks by having them taste and copy her dishes. The juniors will not invent new recipes like her, but they can serve her menu almost as well and much more cheaply.

## Keep in mind

- A small student model learns to imitate a large teacher model.
- The teacher's probabilities or generated text become training data.
- Result: faster, cheaper inference that can run on small devices.
- Students rarely beat teachers, and some providers forbid distilling their outputs.

## Go deeper

- [Knowledge distillation on Wikipedia](https://en.wikipedia.org/wiki/Knowledge_distillation)
- [Distilling the Knowledge in a Neural Network (Hinton et al., 2015)](https://arxiv.org/abs/1503.02531)

## Related terms

- [Supervised and unsupervised learning](https://tidbit.legios.com.ar/supervised-and-unsupervised-learning/): Two ways a machine learns: from examples with answers, or by finding patterns on its own.
- [LLM](https://tidbit.legios.com.ar/llm/): A neural network trained on huge amounts of text to predict the next word.
- [Training and inference](https://tidbit.legios.com.ar/training-and-inference/): Training builds the model once; inference is using it, millions of times a day.
- [DeepSeek](https://tidbit.legios.com.ar/deepseek/): A Chinese AI lab whose open-weight models showed that top AI could be built for less.
- [Open-weights models](https://tidbit.legios.com.ar/open-weights-models/): AI models whose trained numbers anyone can download and run on their own machines.
- [Reasoning model](https://tidbit.legios.com.ar/reasoning-model/): An AI model that works through a problem step by step before it answers.
- [Laptop](https://tidbit.legios.com.ar/laptop/): A whole computer folded into one portable case that runs on a battery.