Word embeddings
The conceptWhy "king − man + woman" lands next to "queen"
A word embedding is a list of numbers, a vector, standing for one token. The model stores one row per token in an embedding matrix and swaps each token ID for its row. Similar words get similar rows.
Think of each vector as a point in space; word2vec used 300 numbers per word. "Cat" sits near "kitten" and far from "invoice", and the step from "man" to "woman" roughly matches "king" to "queen".
Closeness is usually measured by cosine similarity, the angle between two vectors: pointing the same way scores 1, unrelated scores near 0, opposite scores −1.
What it’s forMeasures how alike two words are by the angle between their embedding vectors.
cos(a, b) = (a · b) / (|a| × |b|)
- a, b
- the embedding vectors of two words, e.g. 300 numbers each
- a · b
- dot product: multiply matching numbers and add them up
- |a|
- length of a: square root of the sum of its squared numbers
What it givesA score from −1 to 1; near 1 means the words are used in very similar ways.
- a = (3, 4), b = (4, 3): a · b = 12 + 12 = 24
- |a| = 5, |b| = 5, so cos = 24 / 25 = 0.96
- Almost the same direction: very similar words