Citation accuracy measures whether the sources a generated answer cites actually support the claims they are attached to. In AI applications that show sources to users, citation accuracy tells you whether a reader who opens a link will find the claim backed up.
For example, suppose an assistant writes: “Refunds are issued within five days [1]. Premium support is included at no charge [2].” Source [1] says five business days, so the first claim is only partly supported. Source [2] is a pricing page that lists premium support as a paid add-on, so it contradicts the second claim.
Both links resolve to real pages. A checker that only confirms the links work would mark both citations correct.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why citation accuracy matters
A correctly formatted citation can still be wrong. An answer can cite a real document that says something different from the sentence it is attached to.
Citation accuracy helps you with:
- Catching misleading sources: Find citations that look authoritative but do not back the claim.
- Finding uncited claims: Spot material claims a reader might rely on that have no source at all.
- Locating the failure: Tell whether the problem is a broken link, an unsupported claim, or a missing citation.
- Tracking improvements: Compare prompt or retrieval changes on the same set of judged answers.
Unsupported claims with a citation attached can be harder to spot than an AI hallucination with none, because the citation signals that someone checked.
How citation accuracy works
There is no single universal formula. A complete check separates three questions, because an answer can pass one and fail another.
Source validity: Does the cited document, page, or passage exist, and is it the one referenced?
Claim support: Does the source support the claim as written, including numbers, dates, and qualifiers?
Citation completeness: Do the claims that need support have citations at all?
An answer that cites only its safest sentence can score well on support while failing completeness. That is why you report the three checks separately.
Which claims need citations
Material factual claims need citations. These are claims a reader might rely on, such as figures, dates, eligibility rules, prices, and causal statements.
Opinions and restatements of the user’s question usually do not.
How to measure citation accuracy
Start with a small set of real answers and the context that was retrieved for each one. Then label every claim.
For each claim, record the cited passage and one label:
- Supported: The source backs the claim as written.
- Partly supported: The source backs part of the claim, or a looser version of it.
- Unsupported: The source does not address the claim.
- Contradicted: The source says something different.
Guidance on agent evals from traces shows how to turn real answers into repeatable cases.
Report each component with its denominator
Report each check as its own ratio, and name the denominator every time. Two common ratios are:
claim support = fully supported cited claims / judged cited claims
completeness = cited material claims / all material claims
Where:
- Judged cited claims are the claims that have a citation and received a label.
- Material claims are the claims a reader might rely on, cited or not.
Also report the unit you judged. That might be a claim, a citation, an answer, or a document.
Score the example in Python
The following script scores the refund example. It adds a third claim, “You can cancel any time,” that has no citation.
claims = [
{"claim": "Refunds are issued within five days",
"cited": True, "source_resolves": True, "label": "partly supported"},
{"claim": "Premium support is included at no charge",
"cited": True, "source_resolves": True, "label": "contradicted"},
{"claim": "You can cancel any time",
"cited": False, "source_resolves": None, "label": None},
]
cited = [c for c in claims if c["cited"]]
valid = [c for c in cited if c["source_resolves"]]
supported = [c for c in cited if c["label"] == "supported"]
partly = [c for c in cited if c["label"] == "partly supported"]
print(f"Source validity: {len(valid)}/{len(cited)} citations resolve")
print(f"Claim support: {len(supported)}/{len(cited)} cited claims fully supported")
print(f" {len(partly)}/{len(cited)} cited claims partly supported")
print(f"Completeness: {len(cited)}/{len(claims)} material claims cited")
Output:
Source validity: 2/2 citations resolve
Claim support: 0/2 cited claims fully supported
1/2 cited claims partly supported
Completeness: 2/3 material claims cited
Source validity is perfect, yet no cited claim is fully supported and one material claim has no citation. A single blended score would hide that pattern.
Worked example: what each check catches
The same three claims, side by side:
| Claim | What the source says | Label |
|---|---|---|
| “within five days” | “five business days” | Partly supported |
| “included at no charge” | “paid add-on” | Contradicted |
| “cancel any time” | No citation | Completeness gap |
A link checker catches none of these problems. A claim-level review catches all three.
The review also points to a fix for each one. Tighten the wording of the first claim, retrieve the add-on price for the second, and add a source or remove the third.
Limitations of citation accuracy
- Support is a matter of degree: A source may support a trend but not an exact figure.
- Sources change: A page can change after retrieval, so a later check may disagree with the original.
- Some sources cannot be rechecked: Paywalled or dynamic pages may not be available to the reviewer.
- Judges disagree: Human and model-based judges both disagree on borderline cases.
To manage these, define your labels, keep examples of each, and recheck agreement when the domain changes.
Citation accuracy vs. faithfulness and relevance
These metrics answer different questions about a retrieval-based answer:
- Context relevance: Was the retrieved context useful for the query?
- Faithfulness: Does the answer stay within its provided context?
- Citation accuracy: Is each cited source real, correctly attached, and does it justify this claim?
Citation accuracy adds a reader-facing question to faithfulness. It matters most when sources are links a reader will open.
Grounding and correctness separate another pair of ideas. One asks whether the answer used its context. The other asks whether the answer is true in the world.
RAG evaluation combines these views. Reporting only one number hides which stage failed.
Frequently asked questions
Is there a standard citation accuracy formula?
No. No single standard covers validity, support, and completeness together. State your numerator, denominator, and claim unit, and report the components separately.
Can an LLM judge check citations?
Yes, it can help at scale by comparing each claim with its cited passage. Calibrate it against human-labeled examples and review the cases where they disagree. A judge that never sees the source text cannot check support. For more on this approach, see LLM as a judge.
Does a correct citation prove the answer is true?
No. It shows the claim was judged to be supported by that source, and that judgment can be wrong on borderline cases. If the source itself is outdated or wrong, a well-cited claim can still be incorrect.
Which claims in an answer need citations?
Material factual claims a reader might rely on. These include figures, dates, eligibility, prices, and causal statements.
How is citation accuracy different from agent evaluation?
Agent evaluation scores the wider task. Citation accuracy should sit beside that task score, not replace it. For the broader picture, see agent evaluation.
Putting citation accuracy to work
Pick a sample of real answers with their retrieved context. Label every material claim for support, and note which claims have no citation.
Report source validity, claim support, and completeness as separate ratios, each with its denominator and unit. Sample and adjudicate borderline cases, because automated support checks need human calibration.
Keep the cited passage with each judged claim. In Arize AX, you can attach these labels to the traced answers, so a reviewer can see the claim, the source, and the verdict together.
