AI evaluation is the practice of measuring whether an AI system produces the output you need, on data you care about, at a rate you can defend. In classic machine learning it usually means scoring predictions against labels: accuracy, precision, recall, RMSE. In LLM and agent systems there is often no label, the output is free text or a sequence of tool calls, and the thing you actually care about is whether the user got what they came for.
Scope is the first decision, and most confusion comes from skipping it. Evaluating a foundation model on a public benchmark measures capability. Evaluating your support agent in production measures system behavior: your prompts, your retrieval, your tools, your users. A model that tops a leaderboard can still fail badly inside your application, because your application is not the benchmark.
The failure mode worth naming up front: a single aggregate score hides a broken slice. An overall pass rate of 0.87 can describe a system that is uniformly decent or one that is excellent for 90% of traffic and completely broken for the 10% who ask about refunds. The average is the same. The support ticket volume is not.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Evaluation splits along four axes: offline versus online, ground truth versus reference-free, code-based versus LLM as a judge, and span level versus session level. Most disagreements about “how to evaluate AI” are really disagreements about which cell of that grid you are in.
- Benchmarks measure model capability. Evals measure your system. Both are useful, and neither substitutes for the other.
- An LLM judge is itself a model that needs to be evaluated. Benchmark it against human labels and report its precision and recall before you trust its scores.
- Aggregate scores hide slice failures. Always break results down by input type, user segment, retrieval source, or conversation length.
- For agents, the unit that matters is often the session, not the individual response. A run can have five correct steps and still fail the user.
Offline and online evaluation
Offline evaluation runs against a fixed dataset before anything ships and can gate a deploy.
Online evaluation runs against production traffic on inputs you did not anticipate, at a cost per scored trace.
Teams that only do offline evaluation ship confidently and get surprised. Teams that only do online evaluation find problems after users do. You need both, with CI gating on the offline side and sampled judges on production traffic.
Ground truth, reference-free, code, and judges
With ground truth, you have a correct answer to compare against: a label, a golden response, an expected SQL query, an expected tool sequence. Scoring is straightforward and cheap. Getting the labels is the hard part, and stale labels quietly rot.
Reference-free evaluation scores an output on its own terms, using the input and any supporting context. Faithfulness to retrieved documents, relevance to the question, adherence to a format, presence of a refusal. Most production evaluation is reference-free because production has no answer key.
Code-based checks are deterministic: valid JSON, required fields, latency limits, tool argument types. Use them for everything they can cover.
LLM as a judge handles the rest. Calibrate it against human labels first, using the guide to LLM as a judge.
Span level versus session level
An LLM application produces a tree of operations. Retrieval, one or more model calls, tool invocations, and a final response. Span-level evaluation scores one of those operations: was the retrieved context relevant, did this tool get the right arguments. It tells you where a failure happened.
Session-level evaluation scores the whole interaction: did the user’s problem get resolved, how many turns did it take, did the agent loop. This is the unit that maps to what a user experienced, and it is where agent evaluation increasingly lives. The full picture of connecting traces to scores is in agent evaluation.
Both levels are necessary. Span scores tell you what to fix. Session scores tell you whether fixing it mattered.
Where evaluation programs go wrong
Trusting an unvalidated judge. If you have not measured your judge against a set of human-labeled examples, you do not know your scores are meaningful. Report the judge’s precision and recall on that set, and rerun it when you change the judge prompt or the judge model.
A test set that never changes. A fixed dataset is good for comparison and bad for discovery. If the set has not grown from production failures in three months, it is measuring last quarter’s system.
Evaluating the model instead of the pipeline. In a retrieval system, most quality problems come from what was retrieved, not from generation. Chunk size and retrieval configuration usually move the numbers more than prompt edits do, so score the retrieval step on its own before you rewrite the prompt.
Confusing leaderboard rank with fitness. Public benchmarks are generic by design, and many are saturated or contaminated by training data. They are a shortlist tool, not a decision.
FAQ
What is the difference between an AI evaluation and a benchmark?
A benchmark is a shared, fixed dataset used to compare models against each other on a general capability. An eval is a measurement of your system on your data, against criteria you defined. Benchmarks answer “which model is stronger in general.” Evals answer “does my application work.”
How many examples do I need in an eval set?
Fewer than people expect to start. Fifty to a hundred carefully chosen examples that cover your known failure modes will catch more regressions than a thousand random ones. Grow the set from production failures rather than trying to reach a number.
Can a model evaluate its own output?
It can, and it is often useful, but treat the score as evidence rather than truth. Self evaluation inherits the same blind spots that produced the output. Using a different model as the judge, or a judge with access to context the generator did not have, gives you more independent signal.
What metrics should I use for an LLM application?
Start with correctness or task completion, groundedness against retrieved context, format or schema validity, and refusal or safety rate. Add latency and cost per request, because a system that is right and unusably slow is still broken. Keep the list short enough that every metric has an owner and a threshold.
When should evaluation happen in the development cycle?
Before a change ships, as a gate on a fixed dataset. After it ships, as continuous scoring on live traffic. And whenever something breaks, as the mechanism that turns one angry user report into a permanent regression test. The testing practices around this loop are covered in AI agent testing.
Session evaluation
Session evaluation scores a complete multi-turn conversation as one unit, from the opening message to resolution or abandonment. It asks whether the whole interaction succeeded, not whether any single response was good.
A session includes every user turn, model response, and tool call. It can also contain several traces or agent runs.
Define the session boundaries first
Before you report a session score, state where a session starts and what ends it. A session might start at sign-in, at the first message, or when a task is assigned. It might end at resolution, at a timeout, or at a handoff to a person.
Different boundaries produce different numbers. Two teams scoring the same traffic can disagree simply because they cut sessions differently.
What session evaluation measures
Session evaluation looks at three properties that single-turn checks miss:
- Cross-turn outcome: Did the user get the task done by the end, including any turns spent clarifying?
- Correction recovery: After the system misunderstood or the user corrected it, did later turns use the correction or repeat the error?
- State consistency: Did facts stated early, such as an account number, a date, or a promised refund, stay consistent in later turns?
These properties set session evaluation apart from agent-run evaluation, which scores one run to its own completion. They also set it apart from single-response checks in LLM evaluation.
Example: a six-turn support session
Suppose a support session takes six turns. In turn two, the system states the wrong policy. The user corrects it in turn three. Turns four through six use the corrected policy and resolve the issue.
A turn-level grader marks turn two as failed and the other five as passed. A session grader should record four separate fields:
- Outcome: achieved.
- Corrections required: one.
- Recovery: successful.
- Contradictions after the correction: none.
Report those fields separately. If you average the turn scores into one number, you hide whether the system recovered.
How to run session evaluation
Build session datasets from recorded sessions, each with an expected end state. Grade them with a rubric written for the whole session, not for single turns. For more on building these, see evaluation rubric and evaluation dataset.
For long sessions, inspect the underlying trace for each turn before you trust the session label. Monitor live sessions with AI observability so you can connect each session score back to the turns that drove it.
Ground truth
Ground truth is whatever an evaluation treats as the correct answer for comparison. The label can refer to three different sources, and a score means something different for each.
Three sources of ground truth
A reference answer is written in advance. It is the expected response, label, or extracted fields stored with each example in an evaluation dataset. Reference answers are precise and reusable. They can also be wrong, outdated, or narrower than the full range of acceptable answers.
An observed outcome is what actually happened. Examples include whether the ticket was resolved, the payment completed, or the user accepted the result. Observed outcomes are the strongest evidence of success, but they are the slowest to get.
Observed outcomes are usually available only after deployment. That puts them on the online side of offline vs. online evaluation.
A human judgment is a person scoring or ranking an output against a rubric. It captures quality that has no single correct answer. The cost is that judges disagree, and you need to measure that disagreement too. For more, see human evaluation and human labeling.
Say which source a score used
A claim like “90% against ground truth” is hard to interpret on its own. The reader needs to know whether the comparison used a stored reference, a real outcome, or a judge’s opinion.
For agent work, also say how correctness was judged. It might cover only the final answer, or it might also cover the steps recorded in the trace.
Reference-free evaluation
Reference-free evaluation scores an output without a pre-written correct answer for that input. It is the main option for open-ended production traffic.
The evaluator works from the input, the output, and stated criteria. It might be a rubric-based judge, a set of checks on the output itself, or a person rating quality directly. Typical questions include:
- Is the answer responsive to the request?
- Is it supported by the retrieved passages?
- Does it follow the required format?
Most production requests have no reference answer, and writing one for every request is not practical. Arize’s LLM-as-a-judge guide and agent evals from traces walk through judge-based scoring of real outputs.
Reference-based evaluation
Reference-based evaluation compares the output against a stored reference. The reference is an expected answer, label, or outcome attached to a case in an evaluation dataset.
It is more repeatable and usually cheaper per run. That makes it a good fit for regression testing on a fixed set. Its limits are that it can only score what the dataset covers, and it can penalize correct answers phrased differently from the reference.
How the two modes fail
Each mode fails in its own way, so name the mode wherever you report a score.
- Reference-free scores: They drift when the judge, rubric, or prompt changes. They can also reward fluent, confident text that is not supported. Check them against periodic human evaluation on a sample, and check judge agreement before trusting a trend.
- Reference-based scores: They go stale. A reference that was right last quarter may be wrong now, or it may capture only one of several acceptable answers.
Many teams run both. They use reference-based checks on a curated set before release and reference-free scoring on sampled live traces after release. Every reported number should say which mode produced it.
For the mechanics of both modes, see LLM evaluation.
