Evals are usually grouped by how the output gets graded, and four types cover most LLM applications. Code-based checks and reference-based scores are cheap and mechanical. LLM-as-judge and human review cost more, and they grade what fixed rules cannot express.
Type
How it grades
Best for
Cost
Code-based checks
fixed rules: JSON schema, regex, exact match, does it compile
format, tool arguments, banned strings
free, instant
Reference-based scores
compares output to a known correct answer by exact match, F1 or embedding similarity
classification, extraction, short factual answers
cheap
LLM-as-judge
a model scores the output against a rubric
open-ended prose, tone, faithfulness
cents per row
Human review
people label a sample by hand
ambiguous cases, ground truth
slow, expensive
Run them cheapest first. Deterministic checks run on every commit and catch most breakage.
Word-overlap scores like BLEU or ROUGE are weak for open-ended answers, since many correct wordings score low. Human labels are the reference that shows whether the cheaper graders agree with people.
Real suites mix all four types over the same dataset.
Rewriting in plainer words…
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why are BLEU or ROUGE a poor fit for grading free-form chatbot answers?Word overlap punishes correct paraphrases and can reward wrong answers that share words. Name what you'd use instead, such as a rubric judge.
I group evals by how the output gets graded, and four types cover most LLM apps: code-based checks, reference-based scores, LLM-as-judge, and human review.
Code checks apply fixed rules like a JSON schema, a regex or exact match, so they fit format and tool arguments, and they're free and instant.
Reference-based scores compare the output to a known correct answer with exact match, F1 or embedding similarity, which works for classification, extraction and short facts.
For open-ended prose, tone or faithfulness I use an LLM judge with a rubric, and I keep human review for ambiguous cases and for building ground truth.
I run them cheapest first, with deterministic checks on every commit, and a real suite mixes all four over the same dataset, with human labels showing whether the cheaper graders agree with people.
One thing I avoid is BLEU or ROUGE for open-ended answers, because many correct wordings score low on word overlap.
Weak answers to avoid
Lists benchmarks like MMLU instead of eval typesPublic benchmarks grade models, not your product. The question asks how you grade your app's outputs, so group the types by grading method.
Treats LLM-as-judge as the only evalA judge costs money on every row. Say that cheap code checks catch most breakage first, and judges cover what fixed rules can't express.
Uses BLEU or ROUGE for open-ended answersWord overlap scores many correct wordings low. For free-form prose, use a rubric-based judge and check it against human labels.
Names the types but never says when to use eachA bare list sounds memorized. Tie each type to what it's best for and what it costs, and say you run them cheapest first.
Interview lens
Likely follow-ups, what you can say, and the weak answers to avoid.
