Eval Sets
You change a prompt, try three questions, and the answers look better. But did the change really help? An eval set is a fixed list of test questions, with a way to mark each answer. It works like a mock test before the board exam, so every change gets a score you can compare.
New words
- Eval set
- A fixed list of test cases used to score your AI system.
- Grader
- The method that gives each answer a pass or fail, or a score.
- Regression
- When something that worked before breaks again after a change.
- Margin of error
- How far the true result could be from the number you measured.
- Slice
- The results for one group of cases, such as one category.
- Overfitting
- Tuning so much to one test that it works only on that test.
- Held-out set
- A second test that you keep aside and run only now and then.
What you will learn
- The problem: checking by feel is not enough
- How an eval set works
- Try it: did my prompt change help?
- How many cases? Noise and other traps
- Good habits in production
- In code
- In a real system
- Interview tip
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.