# Loss function

> The single number that says how wrong a model is, so training can shrink it.

- Level: Advanced
- Topics: AI
- URL: https://tidbit.legios.com.ar/loss-function/
- In Spanish: [Función de pérdida](https://tidbit.legios.com.ar/es/loss-function/)

A **loss function** turns a model's mistakes into one number. You show the model an example, compare its answer with the right one, and the loss says how far off it was: zero for a perfect answer, larger the worse it gets. Training is, quite literally, the search for weights that make this number small across millions of examples.

Which formula you use depends on the task. To predict a quantity, like the price of a house, a common choice is **mean squared error**: the average of the squared differences, which punishes big misses much more than small ones. To pick between categories, the usual choice is **cross-entropy**: the model gives a probability to each option, and the loss is high when it gave little probability to the right one. An LLM is pretrained with cross-entropy on one question, asked over trillions of tokens: how much probability did you give to the token that actually came next?

The loss is also what gradient descent needs. Backpropagation works out how the loss changes when each weight moves a little, and gradient descent nudges every weight in the direction that lowers it. That only works if the loss changes smoothly, which is why models are not trained directly on accuracy (a count of right and wrong answers that jumps in steps) but on a smooth stand-in for it.

Watching the loss is how people monitor training. A training loss that keeps falling while the loss on held-out validation data starts rising is the classic sign of overfitting. And a low loss is not the same as a useful model: it only measures what the formula measures. If the formula rewards the wrong thing, the model gets very good at the wrong thing.

## An analogy

A golf scorecard. It does not tell you how to swing; it only counts strokes. But because there is one clear number, you can try changes and keep the ones that bring it down.

## Keep in mind

- The loss squeezes a model's errors into one number; training tries to lower it.
- Mean squared error is common for numbers, cross-entropy for categories and next tokens.
- It has to change smoothly so backpropagation and gradient descent can follow its slope.
- Training loss falling while validation loss rises is a sign of overfitting.

## Go deeper

- [Loss function on Wikipedia](https://en.wikipedia.org/wiki/Loss_function)
- [Linear regression: Loss (Google Machine Learning Crash Course)](https://developers.google.com/machine-learning/crash-course/linear-regression/loss)
- [What is a loss function? (IBM)](https://www.ibm.com/think/topics/loss-function)

## Related terms

- [Training and inference](https://tidbit.legios.com.ar/training-and-inference/): Training builds the model once; inference is using it, millions of times a day.
- [Model parameters](https://tidbit.legios.com.ar/model-parameters/): The billions of numbers a model learns in training; '70B' means 70 billion of them.
- [LLM](https://tidbit.legios.com.ar/llm/): A neural network trained on huge amounts of text to predict the next word.
- [Token](https://tidbit.legios.com.ar/token/): The small chunk of text a language model actually reads.
- [Gradient descent](https://tidbit.legios.com.ar/gradient-descent/): The method that tunes a model's weights by taking small steps downhill on its error.
- [Backpropagation](https://tidbit.legios.com.ar/backpropagation/): The algorithm that works out how much each weight contributed to a network's error.
- [Overfitting](https://tidbit.legios.com.ar/overfitting/): When a model memorizes its training examples instead of learning the pattern.