# Gradient descent

> The method that tunes a model's weights by taking small steps downhill on its error.

- Level: Advanced
- Topics: AI
- URL: https://tidbit.legios.com.ar/gradient-descent/
- In Spanish: [Descenso de gradiente](https://tidbit.legios.com.ar/es/gradient-descent/)

Training a neural network means finding weights that make its mistakes small. Those mistakes are summed up by a **loss function**: one number that says how wrong the model is on a batch of examples. **Gradient descent** is the standard way to lower it. The **gradient** is the list of partial derivatives of the loss with respect to every weight, and it points in the direction where the loss grows fastest. So you step the opposite way: `w = w - lr * grad`, for every weight, then repeat, often millions of times.

The **learning rate** (`lr`) sets the step size and is the most sensitive knob. Too large and the loss bounces around or blows up; too small and training crawls. It is common to warm it up at the start and lower it as training goes on. Computing the exact gradient over the whole dataset at every step would be far too slow, so **stochastic gradient descent** (SGD) estimates it from a small random mini-batch. The estimate is noisy, but cheap, and the noise even helps the model avoid getting stuck in poor regions.

Large models rarely use plain SGD. Optimizers like Adam and AdamW keep running averages of past gradients (momentum) and scale each weight's step by how large its gradients have been, which makes training far less fussy. That bookkeeping costs memory: AdamW stores two extra numbers per weight, one reason training needs several times the memory of simply running the model.

A large network's loss landscape is not a neat bowl. It has an enormous number of dimensions, flat plateaus and saddle points, and gradient descent gives no guarantee of finding the lowest point. In practice it reliably finds weights that work well on new data, which is why it trains essentially every modern model. The gradients themselves come from backpropagation; gradient descent decides what to do with them. The idea is old: the mathematician Augustin-Louis Cauchy described it in 1847.

## An analogy

Walking down a mountain in thick fog. You cannot see the valley, but you can feel which way the ground tilts under your feet, so you take a step downhill, feel again and repeat. Long strides get you down faster, but they can also carry you across the valley and up the other side.

## Keep in mind

- Update rule: each weight minus the learning rate times its gradient.
- SGD estimates the gradient on small random mini-batches.
- Adam and AdamW add momentum and per-weight step sizes, at a memory cost.
- There is no guarantee of the best minimum, but it works well in practice.

## Go deeper

- [Gradient descent on Wikipedia](https://en.wikipedia.org/wiki/Gradient_descent)
- [Gradient descent (Google Machine Learning Crash Course)](https://developers.google.com/machine-learning/crash-course/linear-regression/gradient-descent)

## Related terms

- [Training and inference](https://tidbit.legios.com.ar/training-and-inference/): Training builds the model once; inference is using it, millions of times a day.
- [Neural network](https://tidbit.legios.com.ar/neural-network/): A pile of tiny math units that learns patterns from examples.
- [Model parameters](https://tidbit.legios.com.ar/model-parameters/): The billions of numbers a model learns in training; '70B' means 70 billion of them.
- [Loss function](https://tidbit.legios.com.ar/loss-function/): The single number that says how wrong a model is, so training can shrink it.
- [Learning rate](https://tidbit.legios.com.ar/learning-rate/): The size of the step a model takes each time training adjusts its weights.
- [Regions and Availability Zones](https://tidbit.legios.com.ar/regions-and-availability-zones/): Where AWS physically lives: cities worldwide, each split into separate data centers.
- [Backpropagation](https://tidbit.legios.com.ar/backpropagation/): The algorithm that works out how much each weight contributed to a network's error.