AI evaluation is the practice of measuring whether an AI system produces the output you need, on data you care about, at a rate you can defend. In classic machine learning it usually means scoring predictions against labels: accuracy, precision, recall, RMSE. In LLM and agent systems there is often no label, the output is free text or a sequence of tool calls, and the thing you actually care about is whether the user got what they came for.
Scope is the first decision, and most confusion comes from skipping it. Evaluating a foundation model on a public benchmark measures capability. Evaluating your support agent in production measures system behavior: your prompts, your retrieval, your tools, your users. A model that tops a leaderboard can still fail badly inside your application, because your application is not the benchmark.
The failure mode worth naming up front: a single aggregate score hides a broken slice. An overall pass rate of 0.87 can describe a system that is uniformly decent or one that is excellent for 90% of traffic and completely broken for the 10% who ask about refunds. The average is the same. The support ticket volume is not.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Evaluation splits along four axes: offline versus online, ground truth versus reference-free, code-based versus LLM as a judge, and span level versus session level. Most disagreements about “how to evaluate AI” are really disagreements about which cell of that grid you are in.
- Benchmarks measure model capability. Evals measure your system. Both are useful, and neither substitutes for the other.
- An LLM judge is itself a model that needs to be evaluated. Benchmark it against human labels and report its precision and recall before you trust its scores.
- Aggregate scores hide slice failures. Always break results down by input type, user segment, retrieval source, or conversation length.
- For agents, the unit that matters is often the session, not the individual response. A run can have five correct steps and still fail the user.
Offline and online evaluation
Offline evaluation runs against a fixed dataset before anything ships and can gate a deploy.
Online evaluation runs against production traffic on inputs you did not anticipate, at a cost per scored trace.
Teams that only do offline evaluation ship confidently and get surprised. Teams that only do online evaluation find problems after users do. You need both, with CI gating on the offline side and sampled judges on production traffic.
Ground truth, reference-free, code, and judges
With ground truth, you have a correct answer to compare against: a label, a golden response, an expected SQL query, an expected tool sequence. Scoring is straightforward and cheap. Getting the labels is the hard part, and stale labels quietly rot.
Reference-free evaluation scores an output on its own terms, using the input and any supporting context. Faithfulness to retrieved documents, relevance to the question, adherence to a format, presence of a refusal. Most production evaluation is reference-free because production has no answer key.
Code-based checks are deterministic: valid JSON, required fields, latency limits, tool argument types. Use them for everything they can cover.
LLM as a judge handles the rest. Calibrate it against human labels first, using the guide to LLM as a judge.
Span level versus session level
An LLM application produces a tree of operations. Retrieval, one or more model calls, tool invocations, and a final response. Span-level evaluation scores one of those operations: was the retrieved context relevant, did this tool get the right arguments. It tells you where a failure happened.
Session-level evaluation scores the whole interaction: did the user’s problem get resolved, how many turns did it take, did the agent loop. This is the unit that maps to what a user experienced, and it is where agent evaluation increasingly lives. The full picture of connecting traces to scores is in agent evaluation.
Both levels are necessary. Span scores tell you what to fix. Session scores tell you whether fixing it mattered.
Where evaluation programs go wrong
Trusting an unvalidated judge. If you have not measured your judge against a set of human-labeled examples, you do not know your scores are meaningful. Report the judge’s precision and recall on that set, and rerun it when you change the judge prompt or the judge model.
A test set that never changes. A fixed dataset is good for comparison and bad for discovery. If the set has not grown from production failures in three months, it is measuring last quarter’s system.
Evaluating the model instead of the pipeline. In a retrieval system, most quality problems come from what was retrieved, not from generation. Chunk size and retrieval configuration usually move the numbers more than prompt edits do, so score the retrieval step on its own before you rewrite the prompt.
Confusing leaderboard rank with fitness. Public benchmarks are generic by design, and many are saturated or contaminated by training data. They are a shortlist tool, not a decision.
FAQ
What is the difference between an AI evaluation and a benchmark?
A benchmark is a shared, fixed dataset used to compare models against each other on a general capability. An eval is a measurement of your system on your data, against criteria you defined. Benchmarks answer “which model is stronger in general.” Evals answer “does my application work.”
How many examples do I need in an eval set?
Fewer than people expect to start. Fifty to a hundred carefully chosen examples that cover your known failure modes will catch more regressions than a thousand random ones. Grow the set from production failures rather than trying to reach a number.
Can a model evaluate its own output?
It can, and it is often useful, but treat the score as evidence rather than truth. Self evaluation inherits the same blind spots that produced the output. Using a different model as the judge, or a judge with access to context the generator did not have, gives you more independent signal.
What metrics should I use for an LLM application?
Start with correctness or task completion, groundedness against retrieved context, format or schema validity, and refusal or safety rate. Add latency and cost per request, because a system that is right and unusably slow is still broken. Keep the list short enough that every metric has an owner and a threshold.
When should evaluation happen in the development cycle?
Before a change ships, as a gate on a fixed dataset. After it ships, as continuous scoring on live traffic. And whenever something breaks, as the mechanism that turns one angry user report into a permanent regression test. The testing practices around this loop are covered in AI agent testing.
