Attention
The conceptHow a model works out what "it" means in a sentence
Attention lets each word pull in meaning from the words that matter to it. In "The animal didn't cross the street because it was too tired", "it" mixes in a lot of "animal" and very little "street".
Each word's embedding is multiplied by three learned matrices to make a query (what it looks for), a key (what it offers) and a value (what it passes on).
"It" scores its query against every key with a dot product (multiply matching numbers, add them up). Softmax turns the scores into weights that add to 1, and "it" becomes the weighted blend of the values.
What it’s forComputes, for every word at once, a new vector that blends in the words it should pay attention to.
Attention(Q, K, V) = softmax(Q·Kᵀ / √d) · V
- Q, K, V
- the queries, keys and values of all the words, one row per word
- Q·Kᵀ
- a table of scores: how well each word's query matches each word's key
- √d
- square root of the vector length d (e.g. 64), which keeps scores from growing too big
- softmax
- turns each row of scores into weights between 0 and 1 that add up to 1
What it givesOne updated vector per word, now carrying meaning from the words it matched.
- q = (1,1,0,0), k₁ = (1,1,0,0), k₂ = (0,0,1,1): scores 2 and 0
- Divide by √4 = 2 → 1 and 0; softmax → 0.73 and 0.27
- New vector = 0.73 × v₁ + 0.27 × v₂