Explain the LLM-as-judge pattern. Why is it so widely used?

beginner

← All AI Evals & Observability questions

Re-explain

LLM-as-judge means asking one model to grade another model's output against written criteria. The judge returns a verdict or score, plus a short reason.

You are grading a support reply.
Criterion: does the reply answer the question using only the given context?
Return JSON: {"verdict": "pass" | "fail", "reason": "<one sentence>"}

It spread because most useful outputs are free text: summaries, answers, tone. Exact-match checks cannot score them. Human review costs money and days. A judge is cheap, returns in seconds, and can run on every commit.

But the judge is a model too. Its score is an estimate, so check it against human labels on a sample before you trust it. It also cannot catch a wrong fact that the criteria never mention.

Rewriting in plainer words…

This answer doesn't lend itself to a diagram - it reads best . No credits were charged.

Why there's no diagram: “”

The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓

The diagram below the answer is the concept . Jump to it ↓

Tailored explanation · switch back to · ·

What should the new diagram focus on?

Guided practice for AI Evals & Observability One question at a time. Answer out loud, get graded, see what you missed.

How well did you know this?

AI:

Saved in this browser - sign in to keep your review list.

How should your speech become text?

Listening… your words appear above as you speak - tap Stop when you're done.

Recording · cr - tap Stop & transcribe when you're done.

Transcribing with AI…

Voice:

Keep going - a few more words and AI can grade it.

Interview lens How interviewers actually use this question

Likely follow-ups

How would you check that an LLM judge agrees with human reviewers before you rely on it? Label a sample by hand, compare verdicts, and measure agreement per class, especially how often it passes outputs that humans failed.
If the judge compares two answers side by side, what bias would you watch for, and how would you correct it? Name position bias. Run each pair in both orders and count a win only when the verdict holds both ways, treating a flip as a tie.
Why give the judge one specific criterion instead of asking for an overall 1-10 quality score? A vague 1-10 score is hard to agree on and hard to act on. A narrow pass/fail criterion is easier to check against human labels and to debug.
Your judge passed a reply that stated a wrong fact. Why could that happen, and how would you close the gap? The criteria never covered that fact. Add a criterion that checks against the source context or a reference answer, or a code assertion for fixed facts.
When would you skip an LLM judge and use a different kind of check? Use code checks when the output is structured or exact, like valid JSON or a required field, and humans for high-stakes calls or new criteria.

What you can say

LLM-as-judge means you ask one model to grade another model's output against criteria you write down, and it gives back a verdict or score plus a short reason.
For example, I'd hand the judge a support reply and ask whether it answers the question using only the given context, returning pass or fail in JSON with one sentence of reasoning.
It spread because most useful outputs are free text, like summaries, answers or tone, and exact-match checks can't score those.
Human review can score them, but it costs money and takes days, while a judge is cheap, answers in seconds, and can run on every commit.
The catch is that the judge is a model too, so its score is an estimate, and I'd check it against human labels on a sample before trusting it.
It also can't catch a wrong fact that the criteria never mention, so what it checks is only as good as the criteria you wrote.

Weak answers to avoid

Treats the judge's score as ground truth The judge is a model and its score is an estimate. Say you'd check it against human labels on a sample before trusting it.
Defines it but never says why it spread The question asks why it's widely used. Name free-text outputs that exact match can't score, and the cost and delay of human review.
Asks the judge a vague 'is this good?' With no written criterion the verdict is noisy and hard to act on. Give one specific criterion and ask for a verdict plus a short reason.
Claims it replaces humans and other checks It misses wrong facts the criteria never mention and needs human labels to validate it. Present it as one check that sits next to code checks and human review.

Interview lens

Likely follow-ups, what you can say, and the weak answers to avoid.

Want a quick review of the fundamentals? See the AI Evals & Observability cheatsheet.

← Back to all AI Evals & Observability questions

Pro · $10/mo

50 of 57 AI Evals & Observability answers are in Pro.

Full answers, code samples, and AI explanations that go simpler or deeper. Cancel anytime.

Full answers + code
AI explanations, simpler or deeper
1,000 AI credits / month
Cancel anytime

Get Pro → See plans & pricing →