Semantic Caching
Many users ask the same question in different words. A semantic cache sees that "When do you open?" and "What are your opening hours?" mean the same thing. It then returns a saved answer and does not call the LLM (the AI model behind a chatbot) at all.
New words
- Semantic cache
- A store of saved answers that matches new questions by meaning.
- Embedding
- A list of numbers that stands for the meaning of a text.
- Cosine similarity
- A score where 1.0 means the same meaning and 0 means unrelated.
- Threshold
- The lowest similarity score at which you accept a saved answer.
- Hit and miss
- A hit reuses a saved answer, and a miss has to call the LLM.
- False hit
- A wrong match: the cache returns an answer that does not fit.
- TTL
- Time to live: how long a saved answer lasts before it expires.
What you will learn
- The problem: same question, different words
- How does a semantic cache work?
- Three kinds of caching, compared
- Try it: find the right threshold
- Where does a semantic cache go wrong?
- How to make it safe in production
- In code
- In a real system
- Interview tip
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.