Continuous evaluation means running evals as an always-on part of development and production rather than only before a release. Offline evals run against curated datasets when something changes. Online evals score production traces or sessions as traffic flows through the system, on a schedule or as records arrive.
The reason to do it is that a pre-release suite can only test the inputs you thought of. Production supplies the ones you did not: a new locale, a customer who pastes 40 pages into the chat, a tool that started returning a different error shape, a phrasing that walks the model into a policy corner. Continuous evaluation is how prompt regressions, retrieval failures, safety issues, and changing user behavior surface within hours instead of becoming invisible product debt.
For agents specifically, it also covers behavior that only exists at runtime. A pre-release dataset rarely contains a 14-step trajectory where the agent loops between two tools. Scoring live sessions does.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Continuous evaluation scores live traffic on an ongoing basis, which catches the inputs a pre-release suite never contained.
- Scoring runs asynchronously off traces, never inside the request path, so measurement does not add user-facing latency.
- Sampling is mandatory at any real volume. Cheap deterministic checks can run broadly; judge models run on a sampled or targeted subset.
- Online scores have no ground truth, so they rely on reference-free scorers and user signals such as retries, escalations, and negative feedback.
- The practice only works if someone acts on the output. Alert thresholds, a review queue, and a path from failure back into a dataset are part of the design.
How it works in practice
The application emits traces. A consumer picks up those traces on a schedule or from a stream, prepares them into evaluation inputs, and runs scorers. Results are written back against the trace so a low score is one click from the actual conversation that produced it. That last part matters more than it sounds: a score with no path back to the trace generates a number and no fix. Wiring tracing and evaluation together for an agent is what makes the results actionable rather than decorative.
Scorer selection differs from offline work because there is no reference answer. What works online:
- Reference-free judges. Groundedness against the retrieved context, policy adherence, tone, refusal correctness, and whether the agent actually completed the requested task.
- Deterministic checks. Schema validity, required disclaimer present, forbidden content absent, tool call arguments well formed. These are nearly free and should run on everything.
- Signals the product already emits. Retry rate, session length, escalation to a human, thumbs-down, abandonment. Weak individually, useful in aggregate, and free.
Scheduled offline runs stay part of the picture. Running the curated suite nightly catches slow-moving changes that online scoring, which has no fixed baseline, cannot separate from a shift in traffic.
Cost, sampling, and what does not scale
Judging every production record with an LLM is the plan teams write down first and abandon within a month. Each judged record adds at least one model call, and multiple scorers multiply that. On a high-volume endpoint this becomes a larger recurring bill than the application itself, which is why it is worth understanding where evaluation costs actually accumulate before choosing a sample rate.
The arrangement that survives contact with a budget looks like this: deterministic checks on 100% of traffic, a stratified sample for judge scoring so small segments stay populated, and targeted scoring where something already looks wrong, such as sessions that hit a retry or an error span. Cheaper judge models handle the broad pass, and a stronger model re-scores only what the cheap pass flagged. Record the sample rate per segment, because an aggregate computed across uneven rates is misleading.
Latency is the other constraint. Scoring belongs off the request path entirely. If a decision has to happen before the user sees the response, that is a guardrail with a millisecond budget, not continuous evaluation.
How it relates to the rest of the stack
Continuous evaluation is a practice. It runs on top of a pipeline, executes through a harness, and produces scores that gating can act on. It differs from gating in the decision it informs: gating blocks a release, while continuous evaluation tells you what is happening to something already released. Teams that fold both into their normal development loop are practicing evaluation-driven development, where the suite grows from production failures rather than from an initial guess.
It is also distinct from monitoring. Monitoring watches operational signals: latency, error rate, token spend, throughput. Continuous evaluation assigns quality scores to behavior. A system can be healthy on every operational metric while giving confidently wrong answers.
Where it goes wrong
Alerting on a noisy aggregate. Judge scores vary between identical runs. An alert on a daily mean fires on noise, trains everyone to ignore it, and then misses the real regression. Alert on sustained movement across windows, or on a specific slice, not on a single point.
Nobody owns the output. Scores accumulate, no one is on the hook to look, and the dashboard becomes decoration. A named owner and a few thresholds beats broad coverage nobody reads.
The evaluation itself going stale. The judge model gets updated underneath you, the rubric stops matching current policy, or the scorers were written for last quarter’s product. That is evaluation drift, and continuous evaluation is more exposed to it than offline testing because it runs unattended for months.
Sampling that hides the tail. Uniform sampling at a low rate leaves your rarest and often riskiest segments with almost no records.
FAQ
What does continuous evaluation for AI agents involve?
Three things running together: traces emitted from the agent, scorers that run asynchronously against sampled traces and sessions, and a defined response when scores move, whether that is an alert, a review queue, or a new test case. For agents, the scorers usually include trajectory-level checks such as whether the correct tool was selected, whether the agent recovered from a tool error, and whether the session ended with the user’s task completed rather than just answered.
How is continuous evaluation different from monitoring?
Monitoring tracks whether the system is running: latency, errors, cost, throughput. Continuous evaluation tracks whether the output is any good. They answer different questions and both are needed, because the failure mode specific to LLM systems is a fully healthy service producing bad answers.
How often should online evaluations run?
Match the cadence to the decision. Hourly or near-real-time batches are useful when you are mid-rollout and want to catch a regression fast. Daily is enough for most quality trend tracking. Weekly is usually too slow to connect a score movement back to the change that caused it, because too many things shipped in between.
Do I still need offline evals if I run continuous evaluation?
Yes. Online scoring cannot tell you whether a proposed change is better, because you cannot run two versions against the same live request. A fixed offline dataset with a stable baseline is what supports a before-and-after comparison, and it is what a release gate needs. The two feed each other: production failures become offline test cases.
