Concept of the day1 / 3
Byte-pair encoding
The conceptWhy AI chatbots count your text in tokens, not words
Before a language model reads your text, it cuts it into tokens: chunks that may be a whole word, part of a word or a single byte. Byte-pair encoding (BPE) is the usual way to decide what those chunks are.
BPE starts with single bytes as tokens, then keeps joining the most common neighbouring pair into a new token. After thousands of merges, 'the' is one token and rare words are a few familiar pieces.
What it’s forPicks the next merge while BPE builds its vocabulary.
(a, b) = argmax count(x, y), then replace every 'a b' with 'ab'
- x, y
- any two tokens that sit next to each other in the training text
- count(x, y)
- how many times that pair appears
- argmax
- take the pair with the highest count
- ab
- the new merged token, added to the vocabulary
What it givesOne new token for the most common pair. Repeating this builds the whole vocabulary.
- Text 'ab ab ab ac' starts as 8 symbols: a b a b a b a c
- Pair counts: (a,b) = 3, (a,c) = 1, so merge a+b into 'ab'
- Result: ab ab ab a c, which is 5 tokens instead of 8