Training and inference
Training builds the model once; inference is using it, millions of times a day.
An AI model has two very different lives. Training is when it learns: the neural network sees enormous amounts of data, makes predictions, measures its errors and adjusts its weights, over and over, for weeks or months on thousands of GPUs. At the end you have a file of numbers, the trained model. It happens once per model version, and it is the expensive part.
Inference is everything after: using the trained model to answer. Every time you send a prompt to Claude or ChatGPT, that is one inference. The weights do not change; the model runs the input through the network and produces an output, token by token. Each one is cheap compared with training, but there are billions of them, so inference ends up being most of the computing bill.
The split explains a lot of what you read in the news. A model's 'knowledge cutoff' is the end of its training data. Fine-tuning is a short extra round of training on specific examples. 'Serving' a model means running inference for users, which is why companies need data centers full of GPUs even after the model is finished.
Example: a model that reads chest X-rays is trained on hundreds of thousands of labeled scans over several weeks. Afterward, in a hospital, it looks at one new scan in under a second. That second is inference.
Keep in mind
- Training adjusts the weights; inference uses them without changing them.
- Training is a huge one-time cost; inference is a small cost repeated billions of times.
- Fine-tuning is a short, targeted extra round of training.
- Both run mostly on GPUs, but small models can do inference on a laptop or a phone.
Go deeper
- 3Blue1Brown: gradient descent, how neural networks learn
- NVIDIA: the difference between training and inference
- Machine learning on Wikipedia
