# Training and inference

> Training builds the model once; inference is using it, millions of times a day.

- Level: Intermediate
- Topics: AI
- URL: https://tidbit.legios.com.ar/training-and-inference/
- In Spanish: [Entrenamiento e inferencia](https://tidbit.legios.com.ar/es/training-and-inference/)

An AI model has two very different lives. **Training** is when it learns: the neural network sees enormous amounts of data, makes predictions, measures its errors and adjusts its weights, over and over, for weeks or months on thousands of GPUs. At the end you have a file of numbers, the trained model. It happens once per model version, and it is the expensive part.

**Inference** is everything after: using the trained model to answer. Every time you send a prompt to Claude or ChatGPT, that is one inference. The weights do not change; the model runs the input through the network and produces an output, token by token. Each one is cheap compared with training, but there are billions of them, so inference ends up being most of the computing bill.

The split explains a lot of what you read in the news. A model's 'knowledge cutoff' is the end of its training data. **Fine-tuning** is a short extra round of training on specific examples. 'Serving' a model means running inference for users, which is why companies need data centers full of GPUs even after the model is finished.

Example: a model that reads chest X-rays is trained on hundreds of thousands of labeled scans over several weeks. Afterward, in a hospital, it looks at one new scan in under a second. That second is inference.

## An analogy

Training is medical school: years, expensive, done once. Inference is every patient the doctor sees after graduating: fast, repeated, and what the school was for.

## Keep in mind

- Training adjusts the weights; inference uses them without changing them.
- Training is a huge one-time cost; inference is a small cost repeated billions of times.
- Fine-tuning is a short, targeted extra round of training.
- Both run mostly on GPUs, but small models can do inference on a laptop or a phone.

## Go deeper

- [3Blue1Brown: gradient descent, how neural networks learn](https://www.3blue1brown.com/lessons/gradient-descent)
- [NVIDIA: the difference between training and inference](https://blogs.nvidia.com/blog/difference-deep-learning-training-inference-ai/)
- [Machine learning on Wikipedia](https://en.wikipedia.org/wiki/Machine_learning)

## Related terms

- [Artificial intelligence](https://tidbit.legios.com.ar/artificial-intelligence/): Getting computers to do things that, until recently, only people could do.
- [Neural network](https://tidbit.legios.com.ar/neural-network/): A pile of tiny math units that learns patterns from examples.
- [Model parameters](https://tidbit.legios.com.ar/model-parameters/): The billions of numbers a model learns in training; '70B' means 70 billion of them.
- [GPU](https://tidbit.legios.com.ar/gpu/): The chip that does thousands of simple calculations at once.
- [Prompt](https://tidbit.legios.com.ar/prompt/): Everything you send to an AI model: the question, the instructions, the context.
- [Claude](https://tidbit.legios.com.ar/claude/): Anthropic's family of AI models, and the assistant app built on them.
- [ChatGPT](https://tidbit.legios.com.ar/chatgpt/): OpenAI's chat assistant, the app that brought AI chatbots to the general public.
- [Token](https://tidbit.legios.com.ar/token/): The small chunk of text a language model actually reads.