Next-token prediction
The conceptWhy a chatbot's long answer is really thousands of tiny guesses
A language model only ever answers one question: given the tokens so far, what token comes next? To write a reply, it predicts one token, appends it to the text, and runs again on the longer text, a loop called autoregressive generation.
For "The cat sat on the", the transformer's last layer gives a score, called a logit, to each of its roughly 100,000 vocabulary tokens. Softmax turns scores into probabilities: " mat" 40%, " floor" 20%, " sofa" 10%.
The model then picks one token from that list. Temperature is a number that reshapes the list first: below 1 it makes the top choice even more dominant, above 1 it flattens the list so unlikely tokens get picked more often.
What it’s forTurns the model's raw scores into the probabilities it samples the next token from, adjusted by temperature.
pᵢ = e^(zᵢ/T) / Σⱼ e^(zⱼ/T)
- pᵢ
- the probability of picking token i next
- zᵢ
- the logit, the model's raw score for token i
- T
- temperature, often between 0 and 2; 1 leaves the scores unchanged
- Σⱼ
- sum over every token in the vocabulary, so the probabilities add to 1
What it givesA probability for every possible next token: sharper when T is low, flatter when T is high.
- Logits 1 and 0, T = 1: e¹ ≈ 2.72 vs 1 → 73% vs 27%
- Same logits, T = 0.5: e² ≈ 7.39 vs 1 → 88% vs 12%
- Same logits, T = 2: e^0.5 ≈ 1.65 vs 1 → 62% vs 38%