Offline evaluation happens before deployment, against a fixed dataset in a controlled environment. You run the candidate system (new prompt, new model, new retrieval config) over a golden set. Score the outputs with assertions, judges, or humans, and compare against a baseline.
Offline evals are reproducible, cheap to repeat, and safe: no user sees a bad output.
Their weakness is coverage; a fixed dataset never fully matches the live input distribution.
Online evaluation happens in production on real traffic. It includes A/B tests comparing variants on business and quality metrics, plus LLM judges scoring sampled live traces asynchronously. It also covers guardrail metrics, user feedback (thumbs, regenerations), and drift monitoring.
Online signals reflect reality, including inputs you never anticipated, but they are noisy, slower to read, and mistakes have user-facing cost.
The two are complementary and form a loop. Offline evals gate releases, while online evaluation finds gaps in that suite. Label useful production failures and add them to the offline set for future releases.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
The diagram below the answer is the concept . Jump to it ↓