Checking RAG Quality
A RAG bot can look fine in a demo and still give wrong answers. To fix it, you must first measure it. This lesson shows how to check two things: did the bot find the right text, and did it answer from that text?
New words
- Hit rate
- The share of questions where the right chunk is in the top k.
- Precision
- Of the chunks you kept, how many are the right ones.
- MRR
- Mean reciprocal rank. It scores how high the first right chunk sits.
- Faithfulness
- Every claim in the answer is supported by the retrieved chunks.
- Answer relevance
- The answer really replies to the question that was asked.
- Test set
- A fixed list of questions with the chunk that should answer each.
- LLM judge
- A second model that reads an answer and scores it.
What you will learn
- The bot that sounded sure
- Two places RAG can fail
- Measuring retrieval: hit rate, precision and MRR
- Measuring generation: faithfulness and relevance
- Build a small test set
- Try it: eight test questions
- Use an LLM as the judge
- Debug in the right order
- Trade-offs and real systems
- Interview tip and key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.