# Learning rate

> The size of the step a model takes each time training adjusts its weights.

- Level: Advanced
- Topics: AI
- URL: https://tidbit.legios.com.ar/learning-rate/
- In Spanish: [Tasa de aprendizaje](https://tidbit.legios.com.ar/es/learning-rate/)

Each step of gradient descent says which way every weight should move to lower the loss function. The **learning rate** says how far. It is one number, chosen by people before training starts (a **hyperparameter**, not something the model learns), and it scales every update: `w = w - lr * grad`.

Getting it right matters more than almost any other setting. Too high, and each step overshoots: the loss bounces around or suddenly shoots up, and training **diverges**. Too low, and the model learns so slowly that you run out of time or budget, or it settles somewhere mediocre. Useful values are small, roughly between 0.00001 and 0.01 depending on the model and the optimizer, and the reliable way to find a good one is to try a few on short runs and watch the loss.

In practice the learning rate is rarely constant. A **schedule** changes it over time: a short **warmup** that ramps it up from near zero, so the first chaotic steps do not wreck the weights, then a slow decay, often shaped like a cosine curve, so the last steps are fine adjustments. Optimizers like Adam also scale the step of each weight on their own, but they still sit on top of one base learning rate.

Fine-tuning usually uses a smaller learning rate than training from scratch. The model already knows a lot, and big steps can erase it; small ones adjust its behavior while keeping most of what it learned. Methods like LoRA, which train far fewer weights, tend to use somewhat higher rates than full fine-tuning.

## An analogy

Parking a car. You approach at a decent speed and slow down as you get close. Keep the speed up to the end and you hit the wall; crawl the whole way and you never get there.

## Keep in mind

- It sets the size of each update: weight minus learning rate times gradient.
- Too high and training diverges; too low and it crawls.
- Schedules warm it up at the start and lower it toward the end.
- Fine-tuning normally uses smaller rates than training from scratch.

## Go deeper

- [Learning rate on Wikipedia](https://en.wikipedia.org/wiki/Learning_rate)
- [Linear regression: Hyperparameters (Google Machine Learning Crash Course)](https://developers.google.com/machine-learning/crash-course/linear-regression/hyperparameters)
- [What is learning rate in machine learning? (IBM)](https://www.ibm.com/think/topics/learning-rate)

## Related terms

- [Gradient descent](https://tidbit.legios.com.ar/gradient-descent/): The method that tunes a model's weights by taking small steps downhill on its error.
- [Loss function](https://tidbit.legios.com.ar/loss-function/): The single number that says how wrong a model is, so training can shrink it.
- [Training and inference](https://tidbit.legios.com.ar/training-and-inference/): Training builds the model once; inference is using it, millions of times a day.
- [Model parameters](https://tidbit.legios.com.ar/model-parameters/): The billions of numbers a model learns in training; '70B' means 70 billion of them.
- [Fine-tuning](https://tidbit.legios.com.ar/fine-tuning/): Extra training that adapts an existing AI model to a specific task or style.
- [LoRA](https://tidbit.legios.com.ar/lora/): A cheap way to fine-tune a big model by training a small add-on instead of every weight.