KV Cache
A language model writes its answer one small piece at a time. The KV cache saves its earlier work, so it does not repeat that work at every step. This is the main reason AI chat answers appear fast.
New words
- Autoregressive
- The model uses its own earlier output as the input for the next step.
- Attention
- The way a token looks back at earlier tokens to find what matters.
- Causal
- A token can look only at earlier tokens, never at later ones.
- KV cache
- A store of saved notes (K and V) about every earlier token.
- Prefill
- The first phase, where the model reads your whole prompt at once.
- Decode
- The second phase, where the model writes the answer one token at a time.
- Memory bandwidth
- How much data can move from memory to the chip in one second.
What you will learn
- How an LLM writes: one token at a time
- Query, Key and Value in simple words
- The waste: doing the same work again and again
- See it: with and without a cache
- Prefill and decode: the two phases
- In code
- The price: memory
- How engineers shrink the cache
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.