Faithfulness measures whether an answer accurately reflects the provided context without adding unsupported claims. Hallucination is the failure mode on the other side of that line: the model fabricates a detail, contradicts the source, or extends past what the evidence supports. Score faithfulness and you are scoring how well the generation step stayed inside the material it was given.
The important limit is in the definition. Faithfulness compares the answer to the context. It does not compare the answer to reality. An answer can be entirely faithful to a document that is out of date, mis-scoped, or simply wrong, and the faithfulness score will be high while the user gets bad information. That is why faithfulness belongs in a set of evals rather than alone.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Faithfulness asks one question: is every claim in the answer supported by the provided context? It says nothing about whether the context was right.
- Hallucination is the inverse label for the same comparison. Most judge prompts can produce either by flipping polarity.
- An answer can be faithful but wrong, and correct but unfaithful. Pair faithfulness with a correctness eval when the cost of a wrong answer is real.
- Faithfulness scored on a whole response hides single fabricated sentences. Decompose into claims when precision matters.
- A low faithfulness score does not tell you whether the fault is retrieval or generation. You need the retrieved context on the span to find out.
Faithfulness, correctness, and grounding
Three terms, three different questions, and conflating them is the most common source of argument in eval reviews:
- Faithfulness asks whether the answer is supported by the context that was provided.
- Correctness asks whether the answer is true, judged against ground truth or a reference answer.
- Grounding is the broader property and practice of tying claims to evidence, including retrieval, citation, and attribution. Faithfulness is how you measure one part of it.
Both off-diagonal cases are real and both show up in production. Faithful but wrong: the retrieved policy document describes last quarter’s refund window, the model repeats it accurately, and the customer is told the wrong thing. Correct but unfaithful: the model answers a general knowledge question from what it learned in pretraining, the answer is right, and nothing in the retrieved context supports it. The second case passes a correctness check and fails a faithfulness check, which is the intended behavior of both metrics, not a bug in either.
How faithfulness is measured
The dominant approach is an LLM judge that receives the context and the answer and returns a label. It performs better than most eval tasks because it is a comparison, not a recall test. The judge does not have to know the subject, only whether one text supports another.
Two design choices decide how much the number is worth:
Granularity. A single faithful or unfaithful label on a five-paragraph answer is a coarse instrument. One invented statistic in an otherwise accurate summary often gets rounded away, because the response as a whole reads supported. Splitting the answer into atomic claims and judging each against the context catches that case. It costs more tokens and more latency, which is a real tradeoff when you are already watching what evaluation runs cost across a high-volume application.
Rubric specificity. “Is this answer faithful?” is underspecified. Does a reasonable inference from two stated facts count as supported? Does a hedged sentence count? Does background framing that the context does not mention count? If your rubric does not answer those, two human annotators will disagree with each other and the judge will be inconsistent between runs. Write the edge cases into the prompt, then validate the judge against human labels on your own data. Detector accuracy is domain-dependent, which is the premise behind open benchmark work like the LibreEval RAG hallucination dataset.
Where faithfulness breaks down
Long contexts. Judge attention degrades as the context grows. The same answer scored against 2,000 tokens and 30,000 tokens of context can get different labels, and the drift tends to run toward false “supported” calls because verifying absence is harder than spotting a match.
Abstractive summarization. Good summaries compress and generalize. A strict judge flags legitimate compression as unsupported; a loose one waves through real invention. This is the hardest calibration in faithfulness scoring and there is no setting that is right for every use case.
Attribution ambiguity. When the context contains several documents that partly contradict each other, “supported by the context” stops being a yes or no question. Decide in advance whether support by any single retrieved document counts, or whether the answer must be consistent with all of them.
Blaming the wrong stage. A drop in faithfulness is a symptom. Retrieval that started returning off-topic passages produces the same score as a prompt change that loosened the model’s constraints. Looking at real hallucination examples from your own traces, with the retrieved context attached, is faster than reasoning about the aggregate.
FAQ
Is faithfulness the same as hallucination?
They are two labels for the same comparison. Faithfulness is the positive framing, the share of claims supported by the context. Hallucination is the negative framing, the claims that are not. Teams usually track one and report the other as its complement, and the choice is mostly a matter of which direction your dashboards read better.
Can an answer be faithful and still wrong?
Yes, and this is the single most useful thing to know about the metric. If the retrieved context contains an error, an answer that reproduces it faithfully is wrong. Faithfulness audits the generation step. Auditing the source material is a separate job, handled by document freshness checks and correctness evals against ground truth.
How is faithfulness different from relevance?
Relevance asks whether the answer addresses the question. Faithfulness asks whether it is supported by the evidence. An answer can be supported and useless, if it accurately summarizes a passage that has nothing to do with what the user asked. Retrieval relevance is a third thing again: whether the documents you fetched match the query.
What score should trigger an alert?
There is no universal threshold, and picking one before you have looked at your own distribution produces noise. Score a sample of production traffic, look at where the labels actually cluster, and set the bound relative to that baseline. What matters more is watching for change: a stable 0.9 that drops to 0.8 in a day is a stronger signal than any absolute number.
Do I need a reference answer to measure faithfulness?
No, and that is the practical advantage over correctness. Faithfulness needs only the context and the response, both of which you already have in production. Correctness needs a reference, which usually means a curated dataset and human effort.
