LLM-as-a-Judge
You changed a prompt and now have 2,000 new answers to check. Nobody can read them all. LLM-as-a-judge uses a second model, with a clear marking scheme, to grade the answers at scale. It works well. But you must know its biases and check it against human labels first.
New words
- LLM judge
- A model that reads an answer and gives it a grade.
- Rubric
- A marking scheme that says what each score means.
- Pointwise
- The judge grades one answer at a time.
- Pairwise
- The judge sees two answers and picks the better one.
- Bias
- A habit of favouring something for the wrong reason.
- Calibration
- Checking the judge against human labels before you trust it.
- Cohen’s kappa
- A score of agreement that removes the effect of luck.
What you will learn
- The problem: open-ended answers have no single right string
- How it works: three ways to ask
- Writing a rubric the judge can follow
- Try it: judge two answers
- Known biases of LLM judges
- Mitigations and production practice
- In code
- In a real system
- Interview tip
- Key takeaways
10 quiz questions at the end.
The full lesson, with interactive demos and the quiz, opens after you sign in with Google. It is free.