Coverage measures how much of the expected behavior space an evaluation suite actually exercises. The behavior space includes user intents, edge cases, languages, tools, policies, personas, failure modes, and the shape of real production traffic. Coverage is a property of your test set, not of any single response.
Low coverage is dangerous precisely because it looks like success. An eval suite can be green for months because it never tested the cases that break. Every number on the dashboard is real, and none of them describes the traffic that generates your incidents. Coverage is the metric that tells you how much of the map you have looked at, and it is the one most teams never compute.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Coverage is about a set of test cases: does your eval suite span the situations you care about? Completeness is about one response: did it address the whole question?
- A green eval suite with poor coverage is false confidence, not evidence of quality.
- Coverage is measured against an explicit taxonomy of the behaviors you care about. Without that list there is nothing to compute a percentage against.
- Production traffic is the honest source of the taxonomy. Clustering real inputs finds intents nobody thought to write test cases for.
- Coverage and cost pull against each other. More cases and more evaluators per case is more spend, so choose which dimensions get exhaustive treatment.
Coverage is not completeness
These two collapse into each other constantly, and they operate on different units.
Completeness asks about a single response: did the answer include everything the task required? Coverage asks about a collection: do the cases in the evaluation set span the situations that occur in production? You can score perfect completeness on every item in a dataset of fifty English-language single-turn questions and know nothing about how the system handles a multi-turn Spanish conversation with a tool failure in the middle.
The practical version of the distinction: completeness failures show up as a bad score on a case you ran. Coverage failures show up as an incident on a case you never ran.
How to measure coverage
Coverage needs a denominator, and the denominator is a taxonomy of the behaviors that matter. Building it is most of the work.
Start from production, not from imagination. Sample real traffic, cluster it by intent or topic, and count how many clusters your eval set touches. This is where the interesting answers come from, because the clusters nobody anticipated are exactly the uncovered ones. Sampling from live traces is a routine capability of an agent observability platform, and it is much more reliable than a team brainstorming edge cases in a room.
Enumerate the dimensions separately. Intent, language, input length, tool path, user persona, and failure condition are independent axes. Reporting a single coverage percentage across all of them hides the fact that you have 90% intent coverage and 0% coverage of the case where the retrieval service times out. Track coverage per axis.
Add error analysis output back into the set. Every production failure you triage is a case your suite did not have. Adding it, along with the near-miss variants around it, is the cheapest coverage improvement available and it compounds. This is how a small curated dataset stays relevant instead of aging into a regression suite for problems you fixed a year ago.
Do not confuse volume with coverage. Ten thousand near-duplicate cases sampled from your highest-traffic intent is a large dataset with narrow coverage. A few hundred deliberately diverse cases will find more. Public benchmark efforts such as the LibreEval hallucination dataset are built around spanning conditions rather than accumulating volume, and the same principle applies to an internal set.
The tradeoffs nobody mentions
Coverage costs money and time. Each new case is more inference, more judge calls, and more latency in the loop between a change and a verdict. Broad coverage on every dimension with every evaluator on every commit is not affordable at scale, which is one of the real cost decisions in LLM evaluation. The usual resolution is tiering: a small fast set on every change, the full set nightly or before release.
Coverage can be gamed. A percentage against a taxonomy you wrote is only as honest as the taxonomy. If the categories were chosen after looking at what the suite already tests, the number will be high and meaningless. Write the taxonomy from traffic first.
Rare and important are not the same as frequent. Coverage weighted purely by traffic volume will under-test the low-frequency, high-cost cases: the compliance question, the prompt injection attempt, the request that touches a destructive tool. Weight by risk as well as by frequency, and be explicit about which cases are in the set because they are common and which are there because they are dangerous.
Full coverage does not exist. The input space of a language system is unbounded. Coverage is a direction to push, not a target to reach, and any report claiming 100% is measuring against a taxonomy that is too small.
FAQ
How is coverage different from completeness?
Coverage is a property of the evaluation set: does it span the cases you care about? Completeness is a property of one response: did it include everything the task required. Different units of analysis, so a system can be strong on one and weak on the other.
Is this the same as code coverage?
The intuition transfers, the measurement does not. Code coverage has a finite, countable denominator: lines, branches, paths. The behavior space of an LLM application is open ended, so any coverage percentage is measured against a taxonomy someone chose rather than against a complete enumeration. Treat the number as relative progress, not as an absolute.
How many test cases do I need?
Fewer than most teams assume, if they are chosen for diversity. A few hundred cases that deliberately span intents, languages, lengths, and failure conditions will surface more issues than thousands sampled uniformly from your busiest path. Grow the set from triaged production failures rather than by generating variations of what you already have.
How do I find gaps in coverage?
Cluster production traffic and look for clusters with no matching eval case. Compare the distribution of your dataset to the distribution of real inputs on the axes you care about. Then read the incidents from the last quarter and check whether the suite would have caught any of them, which is usually the most uncomfortable and most useful of the three.
Does synthetic data improve coverage?
It can, especially for rare conditions that are hard to collect: unusual tool failures, adversarial phrasings, low-resource languages. The risk is that generated cases cluster around the generating model’s own priors, so they add volume without much diversity. Use synthetic cases to extend a taxonomy built from real traffic, not to replace it.
