Concept of the day1 / 3
Softmax
The conceptWhy a chatbot's odds for every next word add up to exactly 100%
A language model gives every word a raw score, called a logit, which can be negative. Softmax turns these into probabilities: raise e (about 2.718) to each score, then divide by the total.
Take three candidate words with scores 2, 1 and 0. Their exponentials are about 7.39, 2.72 and 1, totalling 11.11, so the odds become about 66.5%, 24.5% and 9.0%. They are all positive and add to 100%.
What it’s forTurns a list of raw scores into probabilities that sum to 1.
pᵢ = e^(zᵢ/T) / Σⱼ e^(zⱼ/T)
- zᵢ
- the raw score (logit) the model gives option i
- T
- temperature, which stretches or squeezes score gaps; usually 1
- Σⱼ
- sum over every option, e.g. every word in the vocabulary
- pᵢ
- the resulting probability of option i
What it givesA probability for each option, larger for higher scores.
- Scores 2, 1, 0 → e² ≈ 7.39, e¹ ≈ 2.72, e⁰ = 1
- Total ≈ 11.11
- Odds ≈ 7.39/11.11 = 66.5%, 24.5%, 9.0%