Concept of the day1 / 3
Gradient descent
The conceptHow an AI model tunes a billion settings without trying them all
Training a model means finding the settings, called weights, that make its mistakes small. The loss L(w) is one number that scores those mistakes on the training examples: 0 is perfect, bigger is worse.
Gradient descent lowers the loss step by step. The gradient ∇L(w) holds one slope per weight: how fast the loss rises if that weight grows. Moving every weight a little against its slope makes the loss fall.
What it’s forUpdates a model's weights, step by step, to make its error smaller.
w ← w − η · ∇L(w)
- w
- the model's weights, the numbers training adjusts
- η
- the learning rate: how big each step is, often between 0.001 and 0.1
- ∇L(w)
- the gradient: the slope of the loss for each weight
What it givesNew weights that make the model a little less wrong than before.
- Toy loss L(w) = (w − 3)², lowest at w = 3; its slope is 2(w − 3)
- At w = 0 with η = 0.1: slope = −6, so w ← 0 − 0.1 × (−6) = 0.6
- Repeat and w moves 0.6 → 1.08 → 1.46…, closing 20% of the gap to 3 each step