Completeness measures whether a single answer or agent run includes all the information or steps the task required. A complete support answer states every condition the policy attaches to the outcome. A complete coding agent change ships the implementation, the tests, and the documentation update when the task called for one.
Completeness is the dimension that catches omission, and omission is the failure mode that correctness cannot see. A response can be entirely true and still leave the user worse off because it dropped the caveat that changes the decision. That gap is invisible to any eval that only asks whether the words on the page are accurate, which is why completeness has to be scored as its own criterion rather than folded into a general quality score.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Completeness is about one response: did it address the whole question and include everything the task required?
- Coverage is about a set: does your eval suite span the cases you care about? Same intuition, different unit of analysis.
- A correct answer can be incomplete. Track both, or omissions will pass as successes.
- Completeness needs an explicit list of required elements. Without one, judges and human annotators score their own expectations and disagree.
- For agents, completeness applies to the trajectory as well as the output. Skipped verification steps are omissions even when the final answer looks right.
Completeness is not correctness, and it is not coverage
Three distinctions that keep eval reviews from going in circles.
Completeness versus correctness. Correctness asks whether what was said is right. Completeness asks whether everything required was said. An answer that gives the correct refund amount and omits the 30-day window is correct and incomplete. The scores should disagree, and if they never do, one of them is not being measured independently.
Completeness versus coverage. Completeness is a property of a single response. Coverage is a property of a test set: whether the cases you evaluate span the intents, languages, tools, and edge cases that show up in production. You can have perfect completeness scores on a dataset that never asks a multi-part question. The unit of analysis is different, and the two words are not interchangeable.
Completeness versus verbosity. Length is not completeness. A model that pads an answer to cover every possibility scores well on a lazy completeness rubric while being worse to read. If your judge rewards more words, you have built a verbosity metric and labeled it something else.
How to make completeness measurable
The metric only works when “required” is written down. Three ways to do that, in rough order of rigor:
Checklist rubrics. Enumerate the elements a good answer must contain for that task type, then score the response element by element. For a refund question that might be: the eligibility rule, the time window, the refund method, and the exception for digital goods. The score becomes the fraction of required elements present, which is interpretable and diffable across model versions.
Reference answers with required spans. When you already maintain a golden dataset, mark which parts of the reference answer are mandatory and which are optional detail. The judge then checks for the mandatory parts rather than comparing full texts, which avoids penalizing a response for phrasing things differently.
Question decomposition. For multi-part user questions, split the question into sub-questions first, then check whether each was addressed. This catches the most common real-world omission, which is a model answering the first clause of a two-clause question and stopping.
All three depend on a rubric specific enough that two annotators reach the same label. That is the hard part, and it is worth testing directly: have two people score the same twenty responses before you scale the eval. When they split, a jury of several judge models will usually split in the same places, which is a fast way to find the ambiguity in your own definition.
Completeness in agent workflows
For an agent, the output is only half of what completeness applies to. The other half is the trajectory: did the run perform every step the task required? An agent that answers a data question without ever calling the query tool may produce a plausible number and has skipped the work. An agent that edits code and never runs the test suite has produced an incomplete change even if the diff is correct.
Scoring this means defining the required steps per task type and checking the trace against them. This is one of the reasons step-level structure matters in harness engineering: if your run is one opaque call, there is nothing to check completeness against. If it is a sequence of named steps with recorded inputs and outputs, the missing step is visible in the trace.
Judge-based completeness scoring has the same caution as every other judge-scored dimension. The model is reading a rubric you wrote, so the score inherits every gap in that rubric. Deciding whether a judge is the right instrument is worth doing per dimension. For completeness against an explicit checklist, a judge does well, because the task is presence detection rather than open-ended assessment.
FAQ
How is completeness different from coverage?
Completeness is about one answer: did it include everything required? Coverage is about a set of test cases: does the evaluation suite exercise the situations you care about? A system can score highly on completeness across a test set that has poor coverage, which feels like passing and is not.
Can an answer be correct but incomplete?
Yes, and it is the main reason to measure completeness separately. Every statement in a response can be true while the response omits a required disclosure, a necessary step, or the condition under which the advice does not hold. Correctness passes, completeness fails, and the user acts on partial information.
How do you evaluate completeness of AI-generated documentation?
Score it against a required-sections checklist rather than a reference document: parameters described, return values described, error cases covered, at least one usage example, breaking changes noted. Presence checks are mechanical and can partly run in code. Whether each section is any good is a separate quality judgment, and mixing the two into one number makes both harder to act on.
Does penalizing incompleteness make models more verbose?
It can, if the rubric rewards content volume rather than specific required elements. Anchor the rubric to a fixed list, and score anything beyond it as neutral. If concision matters for your surface, measure it as its own dimension so the two tensions stay visible instead of being averaged into one score.
