AI Evals & Observability.

An eval, short for evaluation, is a repeatable test that runs your AI system over a fixed set of inputs and scores the outputs. It is measurement, not a one-off demo.

case:   "refund request, order already shipped"
input:  user message + order state
expect: mentions the 30-day window, does not promise a refund
score:  pass / fail

It is called the core discipline because the model itself is bought, not built. Everyone can call the same API. What you own is your definition of "good" for your task, plus proof that a change helped.

Without evals, every prompt or model change is a guess judged by feel. With them, shipping becomes a loop: measure, change one thing, measure again.

Evals also fail differently from normal tests. There is no compiler error. A bad output looks perfectly confident, so someone must define correctness on purpose.

Evals are usually grouped by how the output gets graded, and four types cover most LLM applications. Code-based checks and reference-based scores are cheap and mechanical. LLM-as-judge and human review cost more, and they grade what fixed rules cannot express.

Run them cheapest first. Deterministic checks run on every commit and catch most breakage.

Word-overlap scores like BLEU or ROUGE are weak for open-ended answers, since many correct wordings score low. Human labels are the reference that shows whether the cheaper graders agree with people.

Real suites mix all four types over the same dataset.

A golden dataset is a small, human-checked set of test cases with known-good answers. It is the fixed reference you score every prompt, model, or retrieval change against.

Each row pairs an input with an expected output or grading criteria, plus any context the system needs.

{"id": "ret-014", "input": "Can I return shoes after 40 days?", "context": ["policy: 30-day returns"], "expected": "No, the window is 30 days.", "must_include": ["30"]}
{"id": "ret-015", "input": "my order never showed up", "expected_tool": "lookup_shipment"}

"Golden" means the labels are ground truth written or approved by people, not model output.

Two things matter most. Keep it frozen, or scores stop being comparable run to run. When the product rules change, re-check the labels as a new version, since a stale label fails answers that are now correct.

LLM-as-judge means asking one model to grade another model's output against written criteria. The judge returns a verdict or score, plus a short reason.

You are grading a support reply.
Criterion: does the reply answer the question using only the given context?
Return JSON: {"verdict": "pass" | "fail", "reason": "<one sentence>"}

It spread because most useful outputs are free text: summaries, answers, tone. Exact-match checks cannot score them. Human review costs money and days. A judge is cheap, returns in seconds, and can run on every commit.

But the judge is a model too. Its score is an estimate, so check it against human labels on a sample before you trust it. It also cannot catch a wrong fact that the criteria never mention.

Offline evaluation runs your system against a fixed dataset before you ship. Online evaluation measures the live system on real user traffic after you ship.

Offline gives a fast, repeatable signal with known expected answers, so you can compare two prompts fairly. But a fixed dataset never covers everything real users ask.

Online sees the real mix of user inputs, but usually has no ground truth. You lean on judges, user signals, and product outcomes instead.

They form a loop, not a choice. Ship on offline results, watch online, then feed the failures you find back into the offline set.

Public benchmarks score the model. Your product is the model plus prompt, retrieval and tools. It also parses output and runs on your users' real inputs. A high score does not transfer.

Three concrete gaps:

• Distribution mismatch. MMLU is multiple-choice academic questions across 57 subjects. HumanEval is 164 small, self-contained Python functions. Neither looks like your support tickets or your codebase.
• Contamination and saturation. Benchmark items leak into training data, so scores flatter the model. Strong models now cluster near the ceiling, and the gaps between them sit inside the noise.
• Wrong target. They say nothing about format compliance, tone or refusal behavior. Nor do they measure latency or cost per request.

Use them to shortlist two or three candidates. Then decide with your own eval set on your own traffic.

A trace is the full record of one request through your app, start to finish. A span is one timed step inside that trace, such as a single LLM call or a retrieval query.

Spans nest. Each one has a parent, so a trace forms a tree with one root span. Every span carries a name, start and end times, and a status. Its attributes hold the model, the prompt and response, and token counts with cost.

trace: chat_request (2.4s)  user=u_912  session=s_44
├── span: retrieve_docs     (310ms)  k=5, index=faq_v3
├── span: llm_call          (1.9s)   model=..., in=1200 tok, out=180 tok
└── span: guardrail_check   (60ms)   verdict=pass

This is what makes debugging possible. You see which step was slow, and what the model actually received. You can also attach eval scores to one span or to the whole trace.