KV cache

Why doesn't a chatbot reread the whole chat for every new word?

A model writes one token at a time, and each new token attends to every token before it. A KV cache stores the keys and values already computed for earlier tokens, so each step computes them only for the newest one.

Take a 1,000-token chat. Without a cache, writing token 1,001 means rerunning all 1,001 tokens through every layer. With one, only the new token runs; its query meets the 1,000 stored keys and mixes their values.

The price is memory. For every token, every layer keeps one key vector and one value vector, so the cache grows in a straight line as the chat gets longer.

KV cache

Why the Claude and OpenAI APIs bill a reused prompt at a discount

Anthropic and OpenAI APIs offer prompt caching: when requests share the same long opening, like a system prompt, its keys and values are kept. Later requests reuse them instead of recomputing, and pay less for that part.

Inside a single chat the same thing happens all the time. Serving software such as vLLM is built around managing the KV cache in GPU memory, because that cache, not the model's weights, limits how many conversations fit on one GPU at once.

KV cache

Old words' keys and values are kept; only the new word's are computed

The reuse is exact because attention here is causal: each token only looks at tokens before it. Adding token 1,001 can't change what tokens 1 to 1,000 computed, so their keys and values at every layer stay the same.

Queries aren't kept: an old token's query was used once, for its own output, and the next word needs only the newest token's query. So each step runs one token through the layers, plus one pass over the cache.