Model distillation

Model distillation, or knowledge distillation, is a way to squeeze much of what a large model knows into a smaller one. You take a big, capable teacher model, run it on lots of inputs, and train a small student model to reproduce the teacher's outputs instead of learning only from the original labeled data.

The trick, described in a 2015 paper by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, is that the teacher's full probabilities carry more information than a single right answer. A photo classifier that says "cat 90%, fox 8%, car 0.01%" is also telling the student which mistakes are reasonable. With today's LLMs the same idea often takes a simpler form: the teacher generates many answers, or full step-by-step reasoning, and that text becomes training data for the student. In early 2025, for example, DeepSeek published several small open models distilled from its R1 reasoning model.

The payoff is at inference time. A distilled model has far fewer parameters, so it answers faster, costs less per request and can run on a laptop or a phone. The limits: the student rarely surpasses its teacher, it tends to lose the most when tasks drift away from what it was distilled on, and many commercial providers forbid in their terms using their models' outputs to train competing models.