Speculative Decoding
A big model writes one token per step, and each step is slow. Speculative decoding lets a tiny, fast model guess the next few tokens. Then the big model checks all the guesses at once. The output is exactly what the big model would have written, often 2–3× faster.
New words
- Decode
- The phase where the model writes the answer one token at a time.
- Draft model
- A small, fast model that guesses the next few tokens.
- Verify
- The big model checks the draft guesses in a single pass.
- Acceptance rate (α)
- The share of draft guesses that the big model accepts.
- Memory-bound
- Speed is limited by reading data from memory, not by maths.
- Batch
- Many user requests that the GPU handles together.
What you will learn
- Why decoding is slow
- The trick: guess cheaply, verify in bulk
- Try it: watch the guesses get checked
- Same quality, guaranteed
- Where the draft comes from
- When it helps, and when it doesn’t
- In code
- In a real system
- Interview tip
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.