Speculative decoding

Why a giant AI model is slow at writing but fast at checking

A large language model writes one token (a word or part of a word) at a time. Each token needs a full pass through billions of numbers, and most of that time goes on loading them from memory.

Speculative decoding adds a small, fast draft model. It guesses the next few tokens, and the big model checks all of them in a single pass, keeping each one it would have written itself.

At the first guess it disagrees with, the big model puts in its own token and drops the rest. Then the small model starts guessing again from there.

Speculative decoding

How Google speeds up AI Overviews in Search

Google says it uses speculative decoding to produce AI Overviews in Search faster, with the same answers. Open-source tools that run models, such as vLLM and llama.cpp, offer it too.

It works best on predictable text. After "the United States of", even a tiny model guesses "America". Typical speed-ups are two to three times, and code, full of repeated patterns, gains the most.

Speculative decoding

Checking five guesses costs about the same as writing one token

The big model's slow part is reading its huge set of weights from memory. Once they are loaded, scoring five guessed positions costs almost the same as one, so each right guess is a nearly free token.

The answer doesn't change: every token you see was approved or written by the big model. A wrong guess only wastes the small model's quick work.