Backpropagation

How a network knows which of its billions of weights to blame

Backpropagation works out, for every weight, how much it contributed to the network's error. That number is the gradient: how much the loss (the error score) would change if you nudged the weight, and in which direction.

It uses the chain rule: if a weight changes the output, and the output changes the loss, the two effects multiply. For y = w·x with x = 2, w = 3 and target 4, the output is 6, the loss is 4, and the gradient is 8.

A positive 8 means "raising w makes things worse, quickly", so gradient descent, from the last lesson, lowers w. Backprop does the blaming; gradient descent does the nudging.

Backpropagation

PyTorch's loss.backward() runs it in one line

In PyTorch, used to train many large language models, a training step calls loss.backward(), and that one line runs backprop over the whole network. TensorFlow and JAX do the same; this is called automatic differentiation.

While the model computes its answer, PyTorch records every operation, like a receipt. backward() replays that receipt in reverse, filling in a gradient for every weight; an optimizer then uses them to update the weights.

Backpropagation

Blame flows back layer by layer, so one sweep covers every weight

Backprop starts at the loss and moves backwards. Each layer gets the blame for its output from the layer after it, multiplies by its own local slopes and passes it on. A weight's gradient is that blame times the input through it.

The trick is reuse: the last layer's blame is computed once and shared by every earlier weight. So one backward pass costs about one or two forward passes, however many weights there are.

Nudging each weight and rerunning the network instead would take one full run per weight: billions of runs per update for an LLM. Backprop's price is memory: it stores every layer's forward-pass values to compute the slopes.