Loss function

A loss function turns a model's mistakes into one number. You show the model an example, compare its answer with the right one, and the loss says how far off it was: zero for a perfect answer, larger the worse it gets. Training is, quite literally, the search for weights that make this number small across millions of examples.

Which formula you use depends on the task. To predict a quantity, like the price of a house, a common choice is mean squared error: the average of the squared differences, which punishes big misses much more than small ones. To pick between categories, the usual choice is cross-entropy: the model gives a probability to each option, and the loss is high when it gave little probability to the right one. An LLM is pretrained with cross-entropy on one question, asked over trillions of tokens: how much probability did you give to the token that actually came next?

The loss is also what gradient descent needs. Backpropagation works out how the loss changes when each weight moves a little, and gradient descent nudges every weight in the direction that lowers it. That only works if the loss changes smoothly, which is why models are not trained directly on accuracy (a count of right and wrong answers that jumps in steps) but on a smooth stand-in for it.

Watching the loss is how people monitor training. A training loss that keeps falling while the loss on held-out validation data starts rising is the classic sign of overfitting. And a low loss is not the same as a useful model: it only measures what the formula measures. If the formula rewards the wrong thing, the model gets very good at the wrong thing.