Offline evaluation runs against a fixed dataset outside the production request path. Online evaluation runs on production traffic or live traces. The difference is not which metrics you compute, because the same evaluator can run in both places. The difference is when you evaluate and against what data, and that decides what the resulting score is allowed to tell you.
Offline evals buy control. Same inputs, same evaluator, so when the number moves you can attribute the move to the prompt, model, or retrieval change you just made. Online evals buy realism. Real users, the current retrieval corpus, tools that time out, inputs nobody on the team would have written down. Neither one substitutes for the other, and the space between them is where most production failures live.
Key takeaways
- Offline evaluation trades realism for reproducibility: a fixed dataset makes changes comparable, which is what you need for prompt iteration, model swaps, and regression gates before release.
- Online evaluation trades reproducibility for realism: it runs on production traffic where there is no reference answer, so it relies on reference-free evaluators such as groundedness, retrieval relevance, and tool selection correctness.
- Your offline dataset is a snapshot of the traffic you knew about. Production drifts away from it continuously, so an offline suite that keeps passing is not evidence that quality held.
- Online evals are sampled, not exhaustive, because every judged span costs tokens and time. Sampling rate is the main cost control you have.
- The loop matters more than either half: production failures found online become rows in the offline dataset, which turns a one-time incident into a permanent regression test.
What offline evaluation actually gives you
An offline eval needs three things: a dataset of examples, an evaluator, and a runner. The dataset is usually a mix of hand-written cases, examples pulled from production traces, and edge cases somebody filed after an incident. Those examples often carry a reference answer or an expected tool call, which lets you score against ground truth.
Because the inputs are fixed, offline evals support the workflows that need comparability:
- Prompt and model iteration. Change one variable, rerun the suite, read the delta.
- Regression gates. Run the suite in CI on a pull request and block a merge that drops a named metric below its threshold.
- Experiment comparison. Two configurations scored on the same rows can be ranked.
One caveat about the word reproducible. If your evaluator is an LLM judge, the same input can produce a different label on two runs, even at temperature zero. Offline reproducibility means the dataset is pinned, the evaluator prompt is versioned, and the judge model is a specific version. It does not mean the score is deterministic, so treat small movements between runs as noise until you have enough rows to rule that out.
What online evaluation actually gives you
Online evaluation reads what the system did in production. In practice that means evaluating spans and traces after the request has already been served, which is why online evals are usually built on the same instrumentation you already use for tracing and evaluating agents rather than on a separate data path.
Three things change once you leave the fixed dataset:
There is no reference answer. Nobody wrote down the correct response to a live user’s question. So online evaluators have to score properties that can be judged from the trace alone: is the answer grounded in the retrieved context, did the retrieved chunks actually relate to the query, did the agent pick a sensible tool, did the session reach the user’s goal.
You evaluate a sample. Judging every span with an LLM judge means paying for a second model call on every request, which is why sampling rates and cheaper judge models show up quickly in any discussion of what evaluation actually costs. Sample by slice rather than uniformly if you care about a low-traffic route.
Timing is a design decision. Most online evals run asynchronously after the response is sent, so they add no user-facing latency. Evals that run inline, in the request path, can block or modify a response, but they spend your latency budget to do it. That inline case is closer to a guardrail than to measurement.
Where the gap between them hides failures
A clean offline suite and a healthy dashboard can coexist with users having a bad time. The usual reasons:
- Distribution shift. Your dataset has the questions you anticipated. Production has typos, other languages, pasted documents, and adversarial phrasing.
- The system changed around the model. The retrieval corpus got reindexed, a tool version changed its error format, an upstream service started returning empty results. None of that appears in a dataset of prompts and expected answers.
- Overfitting to the eval set. Iterating against the same 200 rows for two months produces a system tuned to those rows.
- Aggregates hiding slices. A stable average can sit on top of one segment failing badly, which is why slice-level monitoring is a standard part of agent observability rather than an advanced feature.
The fix runs in one direction: online findings feed the offline dataset. A flagged session becomes a new test case with the correct behavior recorded, so the suite grows toward the real distribution instead of drifting away from it.
FAQ
Which distinction best describes the difference between offline and online evaluation?
Offline evaluation scores a fixed dataset outside the request path, before or independent of release. Online evaluation scores real production traffic, usually from traces, after the request has been served. Everything else, including which metrics you use, follows from that.
Do I need both?
If you ship to real users, yes. Offline evals tell you whether a change is an improvement. Online evals tell you whether the system works for the traffic you actually get. Offline alone means you learn about failures from users. Online alone means you detect problems but cannot iterate safely, because you have no comparable baseline to test a fix against.
Can I use the same evaluator offline and online?
Reference-free evaluators such as groundedness or retrieval relevance run in both places, and using the same evaluator version in both is worth doing because the scores become comparable. Evaluators that need a reference answer only work offline, since production traffic has no labeled correct output.
How much production traffic should I evaluate online?
Start with a low sampling rate on high-volume routes and a much higher one, up to everything, on low-volume or high-risk routes. The binding constraint is cost and judge throughput rather than statistics.
Does online evaluation slow down my application?
Not if it runs asynchronously on traces after the response is sent, which is the normal setup. Evaluation only affects user-facing latency when you deliberately put it in the request path to block or rewrite a response, and in that case it needs a hard timeout and a defined fallback.
