Temperature sampling
The conceptWhy ChatGPT gives a different answer when you ask the same thing twice
A language model does not pick one fixed next word. It gives every possible next token (word piece) a score, turns the scores into probabilities, and then draws one at random, like a weighted dice roll.
Temperature is a single number, T, that reshapes those probabilities before the draw. Dividing every score by T before turning scores into probabilities makes the gaps between them smaller when T is above 1 and bigger when T is below 1.
Example: after "The sky is", the model might score "blue" 2 and "grey" 0. At T = 1, "blue" gets about 88%. At T = 2 it drops to about 73%, so "grey" turns up more often.
What it’s forTurns a model's raw scores into probabilities, with temperature controlling how spread out they are.
pᵢ = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)
- zᵢ
- the model's raw score (logit) for token i
- T
- the temperature, usually between 0 and 2; 1 leaves the scores unchanged
- pᵢ
- the chance that token i is picked
- Σⱼ
- sum over every token in the vocabulary, so the chances add up to 1
What it givesA probability for every token: sharper when T is low, flatter when T is high.
- Scores 2 and 0, T = 1: e² ≈ 7.39, e⁰ = 1, so p ≈ 7.39 / 8.39 ≈ 0.88
- T = 2: scores become 1 and 0, e¹ ≈ 2.72, so p ≈ 2.72 / 3.72 ≈ 0.73