Correctness measures whether an output is factually or logically right for the task. In code, that usually means the tests pass. In support, it means the answer matches policy. In retrieval augmented generation, it means the answer agrees with ground truth, whether that ground truth lives in the source documents or in a reference answer someone wrote.
Correctness is the metric everyone assumes they are already measuring and often is not. It is context-dependent by nature: right for this task, under this policy, at this point in time. A generic judge model asked “is this correct?” with no reference answer, no retrieved context, and no rubric is doing something much weaker than the name suggests. It is checking plausibility against its own training data and reporting it as truth.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Correctness asks whether the answer is true, judged against ground truth, a reference answer, or a checkable outcome. It is not the same as being supported by the context.
- An answer can be correct but ungrounded, and faithful but wrong. Those two failures need different fixes.
- Correctness without a reference is plausibility scoring. Say so out loud rather than letting a dashboard imply otherwise.
- Define correctness per task before you score it. “Right” for a code agent, a support bot, and a summarizer are three different predicates.
- Binary pass or fail is usually more useful than a 1 to 5 scale, because partial credit is where annotators and judges diverge most.
Correctness, faithfulness, and grounding
Three adjacent terms that describe different checks. Hold them apart:
- Correctness asks whether the answer is true, measured against ground truth or a reference.
- Faithfulness asks whether the answer is supported by the context that was provided. It is silent about whether that context was right.
- Grounding is the umbrella property and practice of tying claims to evidence, including retrieval, citation, and attribution. Faithfulness measures one part of it.
Both mismatches happen constantly. An answer can be faithful but wrong when the retrieved document is stale and the model repeats it accurately. An answer can be correct but ungrounded when the model answers from pretraining knowledge that the retrieved context never mentioned. If you only track one of these metrics, you will misdiagnose the other failure. A faithful-but-wrong answer is a data problem, usually indexing or freshness. A correct-but-ungrounded answer is a control problem: the system did the right thing for reasons you cannot audit, and next time it may not.
What counts as correct depends on the task
The word carries a different definition in each surface, and the definition has to be written down before anyone scores anything.
Deterministic tasks have real ground truth. Code either compiles and passes tests or it does not. A SQL query returns the expected rows or different ones. An extraction either matches the record or contradicts it. Here correctness is close to free: run the check and read the result. Prefer this whenever the task allows it, because no judge is as reliable as an executed test.
Reference-answer tasks have a curated expected output. The judge compares the response to the reference for semantic equivalence, not string match, since “the refund window is 30 days” and “customers have a month to request a refund” are the same answer. Building the reference set is the cost, and it is the reason teams skip this and then wonder why their correctness numbers feel soft.
Open-ended tasks have no single right answer, and correctness collapses into a rubric. At that point be honest about what you are measuring: adherence to a set of criteria someone wrote, not truth. That is still valuable, but the label should reflect it.
Scoring correctness with a judge
Most teams reach for an LLM judge because references are expensive. That is a reasonable tradeoff as long as the limits are stated. A judge with a reference answer performs well; a judge without one is guessing from priors and will confidently pass fluent, wrong answers in any domain that is thin in its training data. The question of when a judge is the right tool matters more for correctness than for any other dimension, because correctness is the one where the judge’s own knowledge is doing the work.
Three practices make judge-scored correctness defensible:
- Give the judge everything the answerer had, plus the reference. Context, question, response, expected answer. A judge missing the retrieved context cannot distinguish a wrong answer from an unanswerable question.
- Write the rubric to the edge cases. Is a partially correct answer with one wrong detail a fail? Is a correct answer with an unnecessary caveat still correct? Without explicit rules, two annotators disagree, and the judge inherits that ambiguity as run-to-run variance.
- Measure the judge. Label a few hundred examples by hand and check agreement. When borderline cases dominate, running several judges as a jury and looking at where they split is a cheaper way to find rubric holes than another round of debate.
Cost shapes the design too. Reference-based correctness runs well offline against a curated dataset and poorly as an online check on every request, which is one of the practical tradeoffs in what evaluation costs at production volume. A common split is correctness offline on a fixed set, cheaper signals such as faithfulness online.
FAQ
What is the difference between correctness and accuracy?
Accuracy is a classification metric: the share of predictions that match the label across a dataset. Correctness is a per-output judgment about whether one response is right for one task, and it applies to free text where there is no fixed label set. You can aggregate correctness labels into something that looks like accuracy, but the underlying judgment is qualitative.
Can you measure correctness without ground truth?
Not honestly, not in full. Without a reference you can measure proxies: internal consistency, agreement between multiple samples, whether the answer is supported by the retrieved context. Those catch real failures. They cannot tell you the answer is true, and a dashboard labeled “correctness” that is actually running a reference-free judge overstates what the team knows.
Should correctness be binary or a score?
Start binary. Pass or fail forces the rubric to be explicit and makes disagreement visible instead of averaging it away. Graded scales are useful once you have a stable definition and genuinely need to distinguish “mostly right” from “right,” which is rarer than it sounds.
How does correctness relate to completeness?
Correctness asks whether what was said is right. Completeness asks whether everything required was said. An answer that states one true fact and omits the two required caveats is correct and incomplete, and only tracking correctness would call it a pass.
