Prompt Caching
Many requests to an AI model start with the same long text, such as rules or a document. Prompt caching saves the model's work on that shared start, so the next request is cheaper and faster.
New words
- Prefix
- The start of a prompt, from its first character up to a chosen point.
- Cache hit
- The model finds saved work for the prefix and reuses it.
- Cache miss
- No saved work is found, so the model does all the work again.
- Cache breakpoint
- A marker that shows where the part to be cached ends.
- TTL
- Time to live: how long saved work stays in the cache.
- Prefill
- The step where the model reads your whole prompt before it writes.
- Time to first token
- The wait before the first word of the answer appears.
What you will learn
- From KV cache to prompt caching
- The one rule: it must be an exact prefix
- Try it: a Q&A bot over a product manual
- The economics
- Choosing the cache lifetime (TTL)
- In code
- Silent cache killers
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.