Tokenization
The conceptWhy a chatbot reads 'unhappiness' as a few pieces, not eleven letters
Before a language model sees text, a tokenizer cuts it into tokens: common chunks like words, word pieces or bytes. Each token becomes an ID number from a fixed list, the vocabulary, and the network only sees those numbers.
So 'the' may be one token, while 'unhappiness' is split into pieces like 'un' + 'happiness'.
Byte-pair encoding (BPE), the usual recipe, starts from single bytes. It keeps merging the most frequent pair of neighbouring tokens in its training text until the vocabulary reaches a set size.
What it’s forPicks the next new token while building a BPE vocabulary.
(a, b)* = argmax count(a b), then replace every a b with ab
- a, b
- two tokens that sit side by side in the training text
- count(a b)
- how many times a is directly followed by b
- argmax
- choose the pair with the highest count
- ab
- the merged token, added to the vocabulary
What it givesOne new vocabulary entry; repeating it tens of thousands of times gives the full token list.
- Text 'hug hug hugs pug' split into letters
- Pair counts: u g = 4, h u = 3, g s = 1, p u = 1
- Merge u g → ug; next round h ug = 3, so merge → hug