Transformers
The conceptHow one block, stacked 96 times, became GPT-3
A transformer is a neural network built by stacking one block many times. In each block, attention lets every token pull in what it needs from other tokens, then a feed-forward layer (a small neural network) refines each token alone.
In "The bank by the river flooded", attention lets "bank" take in some of "river", and the feed-forward layer turns that into "riverbank, not money". GPT-3 stacks 96 such blocks, each with its own weights.
Attention ignores word order, so a position signal is added to each token's embedding first. Each block also adds a small edit to the vector instead of replacing it. This is called a residual connection.
What it’s forShows how every sub-layer of a transformer block updates each token's vector.
x ← x + f(LN(x))
- x
- one token's vector, e.g. 12,288 numbers in GPT-3
- f
- the sub-layer: attention first, then the feed-forward layer
- LN
- layer normalisation: rescales x so its numbers keep a steady average and spread
What it givesThe token's vector with one more learned edit added on top, ready for the next sub-layer.
- Start: x = [1, 2]
- Attention adds [0.5, −1] → x = [1.5, 1]
- Feed-forward adds [0.2, 0.3] → x = [1.7, 1.3]