home
›
cheatsheets
›
ai_engineering
›
ai_evals_observability
Free to read Every card and diagram, and Quiz me
2
Free account Star cards. Your Quiz me marks are kept, so next time you drill just what you missed
3
Pro All 57 answers on this sheet, My sheet across every topic, and PDFs
A free account keeps your stars and Quiz me marks. Pro adds all 57 answers on this sheet, My sheet across every topic, and PDFs.
/
Read
Quiz me
Comfortable
Compact
0
Click any code to copy it
Done 0 /0
To review 0
On this sheet
01 Evals & graders02 Eval datasets03 LLM as judge04 Traces & monitoring05 Test before you ship06 Roll out & A/B test07 Online evals & feedback08 Drift & debugging09 Guardrails & red-teaming10 RAG, agents & chat
The model An eval scores your system on a fixed set of cases, so you can tell whether a change helped. Grade with code checks first, then reference answers, then an LLM judge , then people. Check the cheaper graders against human labels. Offline evals gate a change before it ships. Online evals score live traffic, usually with no ground truth.A wrong answer still returns 200, so log a trace per request, with a span for each LLM call, retrieval and tool call. Every score is an estimate . Pin the model snapshot, compare case by case, and ignore gaps inside the noise. Offline evals gate each change before it ships. Online evals score the traces and raise alerts. Labeled failures go back into the eval set.
Evals & graders what an eval is, and how to grade the output What an eval is Define an eval A repeatable test that scores outputs on a fixed set of cases.
Say why evals are the core skill Anyone can call the same model. You own the definition of good.
Trap An LLM failure raises no error. A wrong answer looks as confident as a right one.
Offline and online evals Test a change before release Offline eval. Fixed dataset, known answers. It can block a deploy.
Measure the live system Online eval. Real traffic, usually no ground truth. It alerts.
Trap A fixed dataset never covers everything real users ask, so watch production too.
Pick a grader Cheap Fast Prose
Code checks Check format and banned strings
Reference scores Score against a known answer
LLM judge Grade open-ended text like tone
Human review Set the ground truth
Tip Code checks cost nothing and never vary, so run hundreds per change. They can't tell if an answer helps.
Trap BLEU and ROUGE score word overlap. A correct answer in different words scores low.
When people must review Decide what good means Label the first golden cases and write the rubric.
Trust a judge at scale Label a sample and check the judge agrees.
Find failures you haven't seen yet Read raw outputs to see what actually breaks.
Tip Safety, legal and medical calls, and anything that moves money, always get a person.
Trap Human review is a recurring cost. Automated scores are only as good as the labels behind them.
Public benchmarks Say why a high MMLU score doesn't transfer It tests the model on exam questions, not your product or users.
Explain contamination and saturation Test items leak into training, and top models bunch at the ceiling.
Say how LMArena, now Arena, ranks models Blind votes between two replies, fit with a Bradley-Terry model.
Tip Shortlist two or three models with benchmarks. Decide with your own eval set.
Trap Formatting and length win Arena votes. Providers can test variants privately and publish the best.
Eval datasets build, cover and version the cases you score Build a golden dataset {"id": "ret-014", stable id "input": "Return shoes after 40 days?", "expected": "No, it's 30 days.", "must_include": ["30"]} a code check
Define a golden dataset A small set of cases with known-good answers, approved by people.
Build one from production Sample traces by intent, segment and length. Add known failures.
Label the rows Two labelers per row. Their disagreement caps any judge you build.
Tip Drop near-duplicates, and redact PII in the trace store before any export.
Trap Re-label as a new version when rules change. A stale label fails correct answers.
Cover and version the set Define coverage Every important slice has enough cases to trust its score.
Plan coverage Make a grid of the axes that matter. Set a floor, say 20 per cell.
Version an eval dataset Never edit in place. Hash each release, and give cases stable IDs.
Handle a version bump Re-run the current best version for a new baseline.
Tip Keep a production-like mix and an oversampled hard mix, and report them apart.
Trap A label fix makes older scores non-comparable. Log fixes apart from added cases.
Start with no traffic Write the first cases Write 20 to 50 by hand with a domain expert, from tickets and docs.
Name the risks of synthetic data Too clean, too similar, and one model writing, answering and judging.
Tip Then have a model expand the seeds into paraphrases and messy inputs. A person approves every label.
Trap Synthetic cases are a start, never ground truth. Swap in real traffic when it arrives.
Don't overfit the eval set Split the eval set A dev set to tune on, and a locked holdout you open at release.
Keep the set honest Add fresh production samples each cycle.
Spot overfitting The dev score rises while the holdout stays flat.
Tip If the metric moves but online signals don't, the win isn't real. That's Goodhart's law.
Trap Keep few-shot examples and tuning cases out of the eval set entirely.
LLM as judge grade free text with a model, then check the grader How LLM-as-judge works Is every claim in the context? one criterion Context: {chunks} Answer: {answer} List unsupported claims first. reason first {"verdict": "pass" | "fail"} label last
Define LLM-as-judge One model grades another's output against written criteria.
Say why it's so widely used Most output is free text. A judge is cheap and fast enough for CI.
Write a judge prompt Pass or fail labels, one criterion per call, and no model names.
Tip Most judge disagreement comes from a fuzzy criterion, not a weak model.
Trap The score is an estimate. Check it against human labels before you trust it.
Use a rubric, not one score Define rubric-based evaluation Split quality into named criteria, and score each one on its own.
Say why a 1-to-10 score fails Nobody shares its meaning, and scores bunch at 7 and 8.
Combine the criteria Weight them in your own code, not in the model.
Tip Use pass or fail, or three levels. A failing criterion points at the fix.
Pairwise or pointwise Decide whether prompt B beats prompt A Pairwise. Choosing between two is easier, and steady bias cancels out.
Gate CI, catch a weekly drop, or score live traffic Pointwise. Each output gets an absolute score on its own.
Tip Use pairwise to pick the change, and pointwise to track it.
Trap Pairwise gives a win rate, not a level, and ranking many candidates takes many pairs.
Judge biases Fix position bias Judge each pair twice, order swapped. Count only agreed wins.
Test for verbosity bias Pad a good answer with filler, and see if its score rises.
Avoid self-preference bias Use a judge from another vendor, and strip author tells.
Trap Random order only hides position bias. Track how often a swap flips the winner.
Test the judge Measure whether a judge is any good Compare it with human labels on a held-out set.
Pick the agreement metric Cohen's kappa, which removes chance, plus recall on failures.
Keep a judge calibrated over time Pin its version. Re-check it against fresh human labels on a schedule.
Tip Aim for human-to-human agreement, not 100%. Frozen anchor items show when the judge drifts.
Trap Raw accuracy misleads. A judge that always says pass scores 95% if 5% fail.
Traces & monitoring see what the model saw, and what it cost Traces and spans trace chat_request 2.4s span retrieve_docs 310ms span llm_call 1.9s span guardrail_check 60ms
Define a trace and a span A trace is one request. A span is one timed step in it.
Say what an LLM span records Model and prompt version, prompt, response, tokens, cost, timing.
Say why status codes aren't enough A confident wrong answer still returns 200 at normal latency.
Join eval scores to what they judged One trace per task. Eval results carry the trace and span id.
Tip Spans nest into a tree with one root. Scores and feedback attach by trace or span id.
Trap Export spans through a queue. A blocking exporter turns observability into an outage.
What to monitor Monitor the basic signals Traffic, p95 latency, time to first token, and errors by cause.
Track the signals only LLM calls have Tokens per call, retries and fallbacks, and truncated answers.
Report cost so it drives decisions Cost per successful task, joined to latency and pass rate.
Tip Page on tail latency and error rate. Read token spend as a weekly trend.
Trap A cheaper model that fails more isn't cheaper. Divide spend by tasks completed.
OpenTelemetry and platforms Instrument LLM calls without lock-in OpenTelemetry's GenAI conventions. Any backend reads the spans.
Keep traces on your own servers Self-host Langfuse or Arize Phoenix.
Pick a platform for LangGraph, or for fast experiments LangSmith for LangGraph apps. Braintrust for eval-first CI diffs.
Tip Prompt and response capture is off by default. Turn it on, and still redact.
Trap The conventions are still experimental. gen_ai.system became gen_ai.provider.name , so pin versions.
Sensitive data in traces Decide what to log freely Metadata. Versions, tokens, latency, cost, tools and scores.
Handle prompt and response text Mask it in your process at the SDK hook, before export.
Set retention Metrics for years, redacted traces for months, raw text for days.
Trap Detectors miss PII in free text, so short retention is the real backstop.
Test before you ship catch regressions before users do Regression tests in CI Regression test a prompt change Rerun saved cases. Nothing that passed may now fail.
Compare the two runs Case by case. A flat average can hide a broken case.
Store prompts As files in the repo, so a pull request that edits one runs the evals.
Gate the merge On the change against the base branch, with a noise band.
Tip Cheap checks on every push, the judge suite nightly. Cache outputs by prompt hash.
Trap A prompt is one global instruction. Fixing one complaint can change unrelated outputs.
Trust scores from noisy outputs Cut the variance you control Pin the snapshot, set temperature 0, pass a seed, freeze the data.
Handle the variance that's left Run each case k times. Report a mean and a confidence interval.
Compare two variants Paired, on the same cases. Ship only if the gap beats the noise.
Tip Cases whose repeats disagree are your flaky set. Fix the prompt or rubric there.
Trap Temperature 0 isn't deterministic. Batching still changes outputs, and OpenAI's seed is best-effort.
Eval-driven development Define eval-driven development Write the eval first, then build until the score clears a bar.
Name the gates on the way to launch Agreed cases, a trusted judge, no regression, then online metrics.
Decide how much to spend on evals By what a bad output costs. Fund whichever layer is the bottleneck.
Trap A gate nobody can fail only reports. Every gate must be able to block a release.
Migrate off a retiring model Prepare before the old model shuts down Replay the golden set on it now. Cache outputs, scores, cost.
Compare the successor Per case. Many regressions go away when you re-tune the prompt.
Trap If your judge runs on the old model, move and re-validate it first.
Roll out & A/B test try a change on real traffic without hurting users Shadow, then canary Test on real traffic before any user sees it Shadow deployment. It gets real requests, but nobody sees its output.
Test with a small slice of real users Canary release. Watch error rate, latency and refusals.
Keep a chat on one version Route canary traffic by session, not by request.
Tip Shadow has no user signal. Score it with a judge, or diff it against current output.
Trap In shadow, stub tool calls, writes and emails, or side effects fire twice.
A/B test an AI feature Randomize the test By user or session, so answers stay consistent.
Keep the control arm fixed Pin a dated model snapshot. Aliases can change.
Pick the winner On the product outcome, like task completion, not eval scores.
Tip Add cost per task and refusal rate to the guardrail metrics. Noisy output needs more traffic.
Trap Variant outputs leak into logs, retrieval stores and training data, and taint the control.
A/B test pitfalls Analyze a test you randomized by user Cluster the variance by user, or chatty users shrink the error bars.
Compare cost, tokens or latency Use quantiles, a bootstrap or trimming. Means jump on long tails.
Watch results while the test runs Fix the sample size first, or use a sequential test.
Tip Give each arm its own cache namespace, or one arm warms the other.
Trap The best of ten variants overstates its lift. Re-measure it on fresh traffic.
Online evals & feedback judge live traffic, and learn from users Judge and sample live traffic Run an LLM judge on production traffic Async, off the trace stream. All suspicious traces plus 5% random.
Split a small human review budget A fifth at random, for the true error rate. The rest on likely failures.
Quote a quality rate from targeted samples Store each trace's sampling probability p, and weight by 1/p.
Tip Rotate the targeting rules often, or they keep finding the same bug.
Trap Pin the judge's model snapshot, or a score drop may just be the judge drifting.
Thumbs and implicit signals Store a thumbs click On the response's trace id, with a reason tag.
Say why thumbs are a weak signal Few click, angry users click most, and few spot a confident error.
Name implicit signals Copies, edits after accepting, retries, rephrasing, abandonment.
Tip Treat thumbs as a queue for error analysis, not a score. Watch the trend.
Trap Check each signal against human labels. Optimizing acceptance rate breeds short, agreeable answers.
Error analysis Start error analysis Read real traces and note the first failure in each. That's open coding.
Turn notes into failure modes Cluster the notes into named modes and count them. That's axial coding.
Decide what to fix and when to stop Fix in count order. Stop when traces show no new modes.
Explain the data flywheel Failures from traces become labeled cases in a gating eval set.
Tip Write evals only for modes that repeat. Logging plus a dashboard is not a flywheel.
Trap Users who churn send no traffic, so their failures never reach the eval set.
Drift & debugging find out why quality fell Kinds of drift Detect input drift Cluster input embeddings. Watch the population stability index.
Detect a silent model change Run a canary set every few hours. Watch length, refusals, parse errors.
Detect retrieval drift Watch index age, empty hits and top-k similarity scores.
Detect concept drift Relabel a fresh sample and compare with the golden labels.
Respond to a model change Keep the old snapshot routable. Re-run the golden set first.
Trap Most drift is silent. Latency and errors stay green while quality falls.
Debug a bad answer or a quality drop Find the bad answer Look up its trace by the id the feedback button stored.
Read the trace Find the first failure. Check retrieval, the final prompt, then tools.
Replay the logged inputs Same snapshot, several runs. Always fails means inputs, not variance.
Debug a drop your offline evals missed Freeze the rollout. Check logging, sampling and the judge first.
Find where the data differs Look for production clusters thin or missing in the eval set.
Tip Add the failing case to the eval set, and score it over several runs.
Trap A live rerun samples a new output, and the index or model alias may have moved.
Guardrails & red-teaming block bad output, and attack your own app Guardrails and their metrics Tell a guardrail from an eval An eval scores a sample. A guardrail acts on every request.
Track a guardrail Per rule, the trigger rate, false positives and negatives, latency.
Keep regressions visible behind a guardrail Score raw output too, or a drop looks like a rising block rate.
Tip Re-run a fixed red-team set after every rule change.
Trap Trigger rate alone is a vanity number. Label passed traffic too, or you never see false negatives.
Red-team the app Red-team an LLM app Human testers plus an attacker model that mutates known attacks.
Say how safety evals differ from quality evals Attacker-chosen inputs, a growing set, and one success fails.
Report the result Attack success rate, counting any success in k runs.
Tip Rank findings by blast radius, the tools, credentials and data an injection reaches.
Trap Payloads often arrive indirectly, in a retrieved document, web page or tool result.
RAG, agents & chat evaluate systems with more than one step Faithfulness and relevance Define faithfulness and relevance Is the answer backed by the context? Does it answer the question?
Measure faithfulness Supported claims divided by all claims, checked against the context.
Measure answer relevance Generate questions from the answer, and compare them to the real one.
Tip They're independent. A grounded answer can still miss the question.
Trap Faithful isn't correct. If retrieval brings the wrong passage, a faithful answer is still wrong.
Evaluate retrieval alone Check the right chunk came back recall@k, on queries labeled with the chunks they need.
Check it ranked near the top MRR or NDCG. Position changes what the model uses.
Decide which half to fix Chunk missing, fix retrieval. Chunk there, fix prompt or model.
Tip Retrieval checks call no model, so run them on every index change.
Trap Retrieval is a hard ceiling. If the chunk misses the top k, no prompt can fix it.
Evaluate an agent Score an agent Task success rate. Was the goal met at the end?
Measure reliability pass^k, all k runs succeed. pass@k needs just one.
Log cheap signals of user pain Cost per task, tool errors, human takeovers, step-cap hits.
Score the path the agent took Required steps in order, extras allowed. Or a judge reads the spans.
Tip Grade the outcome first. Add path checks for loops or unsafe writes.
Trap Errors compound. 98% per step over 20 steps is 67% end to end.
Evaluate a multi-turn chat Test a whole conversation An LLM plays the user, with a persona and a hidden goal.
Score the session Goal met, turns taken, contradictions and forgotten facts.
Tip Keep per-turn checks too, and cap the turns so runs end.
Trap A fixed script of user turns breaks once your system answers differently.
Evaluate with no right answer Check what's still checkable Format, length, tone and policy. Run these on all traffic.
Judge quality Compare with today's output. Ranking beats scoring.
Watch outcomes Accept rate, edits after accepting, regeneration rate.
Trap BLEU or similarity to one gold text punishes the best of many good answers.
Nothing matches. Try a shorter word, or clear the search.
Practice Guided practice One question at a time, with the ones you miss coming back. Practice 57 interview questions Full answers, from beginner to expert.
Found this useful? Pass it on.
Share on LinkedIn
Share on X
Copy link
This sheet stays free. With Pro , you can:
Every answer on this sheet. Read the full answer to all 57 questions it links, not just the 7 free ones.My sheet. The cards you star and the ones you miss in Quiz me, from every sheet, on one page to quiz and print.PDFs. Download this sheet and My sheet, laid out for A4, with every question a link to its answer.The rest of Pro. Every answer in the question library, and 1,000 AI credits a month.
