# Backpropagation

> The algorithm that works out how much each weight contributed to a network's error.

- Level: Advanced
- Topics: AI
- URL: https://tidbit.legios.com.ar/backpropagation/
- In Spanish: [Retropropagación](https://tidbit.legios.com.ar/es/backpropagation/)

To improve a neural network with gradient descent you need a gradient for every one of its weights: how much the loss would change if that weight moved a tiny bit. With millions or billions of weights, testing them one by one is hopeless. **Backpropagation** computes all of them in one sweep. It is the chain rule from calculus, applied systematically from the output back to the input.

It works in two passes. The **forward pass** runs a batch of examples through the network layer by layer and computes the loss, keeping the intermediate results (the activations) in memory. The **backward pass** starts at the loss and walks the layers in reverse. Each layer receives the gradient with respect to its output, uses its stored activations to compute the gradients for its own weights, and hands the gradient with respect to its input to the layer before it. The backward pass costs roughly twice as much as the forward pass, however many weights there are.

Nobody writes this by hand anymore. Frameworks like PyTorch and JAX record the operations of the forward pass and differentiate them automatically (**automatic differentiation**); calling `loss.backward()` in PyTorch is backpropagation. Those stored activations are a big reason training uses so much more memory than inference, and tricks like activation checkpointing trade extra computation for less memory.

Backpropagation also explains some classic failures. Multiplying many small derivatives through a deep stack of layers can make gradients shrink toward zero (**vanishing gradients**) or blow up; residual connections, normalization layers and careful initialization exist largely to keep that signal healthy. The method was discovered more than once, and a 1986 paper by David Rumelhart, Geoffrey Hinton and Ronald Williams made it the standard way to train neural networks.

## An analogy

A kitchen tracing a bad dish backwards. The diner says it is too salty; the head chef asks the sauce station how much salt it added, the sauce station asks the stock station, and so on down the line, so every cook learns how much of the problem was theirs and adjusts next time.

## Keep in mind

- It is the chain rule applied layer by layer, from output to input.
- The forward pass computes the loss; the backward pass computes every gradient.
- Stored activations make training far hungrier for memory than inference.
- Frameworks do it automatically: `loss.backward()` in PyTorch.

## Go deeper

- [Backpropagation on Wikipedia](https://en.wikipedia.org/wiki/Backpropagation)
- [What is backpropagation really doing? (3Blue1Brown)](https://www.3blue1brown.com/lessons/backpropagation)

## Related terms

- [Neural network](https://tidbit.legios.com.ar/neural-network/): A pile of tiny math units that learns patterns from examples.
- [Gradient descent](https://tidbit.legios.com.ar/gradient-descent/): The method that tunes a model's weights by taking small steps downhill on its error.
- [Model parameters](https://tidbit.legios.com.ar/model-parameters/): The billions of numbers a model learns in training; '70B' means 70 billion of them.
- [Framework](https://tidbit.legios.com.ar/framework/): A ready-made skeleton for an app: it sets the structure and calls your code.
- [Training and inference](https://tidbit.legios.com.ar/training-and-inference/): Training builds the model once; inference is using it, millions of times a day.