Hallucination

Overview

The Hallucination evaluator determines whether an assistant’s response contains claims that are unsupported by, or that contradict, the conversation it had access to. Unlike Faithfulness — which grounds a single response in a single block of retrieved context — Hallucination grounds the response in the broader conversation: earlier user and assistant turns, tool calls, tool results, and any retrieved context.

When to Use

Use the Hallucination evaluator when you need to:

Evaluate multi-turn agents and assistants - Check whether a response invents facts that were never established across the conversation
Catch fabricated tool results - Detect when a response asserts data that a tool never returned (or returned an error for)
Verify grounding beyond a single RAG context - Judge responses against everything the model saw, not just one retrieved document

This evaluator judges the response against the conversation as its source of truth. It is the conversation-level counterpart to Faithfulness; reach for Faithfulness when you have a single retrieved context block and Hallucination when grounding lives across the conversation and tool activity.

Supported Levels

The level of an evaluator determines the scope of the evaluation in OpenTelemetry terms. Some evaluations are applicable to individual spans, some to full traces or sessions, and some are applicable at multiple levels.

Level	Supported	Notes
Span	Yes	Apply to an LLM span, using its message history as the `input` and its latest response as the `output`.
Trace	Yes	Apply across a trace whose spans form the conversation available to the response.
Session	Yes	Apply across a multi-turn session, using the ordered turns as the `input`.

Relevant span kinds: LLM and agent spans, particularly in multi-turn or tool-using pipelines.

Input Requirements

The Hallucination evaluator requires two inputs:

Field	Type	Description
`input`	`string`	The full record the assistant had access to before responding — prior user and assistant turns, tool calls, and tool results, plus any retrieved or provided content — rendered as a single, readable transcript with every turn labeled by its role. Treated as the only source of truth.
`output`	`string`	The assistant’s latest response to classify.

Formatting the input

Pass input as a single, human-readable transcript of the whole record — not raw JSON. Label every turn with its role, and mark tool calls and their results clearly:

User: What's our refund window?
Tool (lookup_policy): Refunds: 30 days from delivery.
Assistant: 30 days from delivery.
User: And for electronics?

Keep the role labels on every turn. The evaluator weighs evidence by its source: user messages, tool results, and retrieved or provided content are treated as authoritative ground truth, while the assistant’s own earlier turns are not counted as independent evidence for a claim. If you strip the roles and pass an unlabeled blob, the evaluator can’t tell an authoritative tool result from an unverified assistant claim, and its grounding judgments degrade. Always keep the role on every turn in the input, and label tool outputs as tool results.

Output Interpretation

The evaluator returns a Score object with the following properties:

Property	Value	Description
`label`	`"hallucinated"` or `"grounded"`	Classification result
`score`	`1.0` or `0.0`	Numeric score (1.0 = hallucinated, 0.0 = grounded)
`explanation`	`string`	LLM-generated reasoning for the classification
`direction`	`"minimize"`	Lower scores are better
`metadata`	`object`	Additional information such as the model name. When tracing is enabled, includes the `trace_id` for the evaluation.

Interpretation:

Grounded (0.0): Every claim in the response restates, or follows necessarily from, the input (ordinary general knowledge is allowed as long as it doesn’t contradict the input)
Hallucinated (1.0): The response asserts situation-specific facts not present in the input, or contradicts it

Usage Examples

Python
TypeScript

from phoenix.evals import LLM
from phoenix.evals.metrics import HallucinationEvaluator

# Initialize the LLM client
llm = LLM(provider="openai", model="gpt-4o")

# Create the evaluator
hallucination_eval = HallucinationEvaluator(llm=llm)

# Inspect the evaluator's requirements
print(hallucination_eval.describe())

# Evaluate a single example
eval_input = {
    "input": (
        "User: What's our refund window?\n"
        "Tool (lookup_policy): Refunds: 30 days from delivery.\n"
        "Assistant: 30 days from delivery.\n"
        "User: And for electronics?"
    ),
    "output": "Electronics can be returned within 90 days.",
}

scores = hallucination_eval.evaluate(eval_input)
print(scores[0])
# Score(name='hallucination', score=1.0, label='hallucinated', ...)

import { createHallucinationEvaluator } from "@arizeai/phoenix-evals";
import { openai } from "@ai-sdk/openai";

// Create the evaluator
const hallucinationEvaluator = createHallucinationEvaluator({
  model: openai("gpt-4o"),
});

// Evaluate an example
const result = await hallucinationEvaluator.evaluate({
  input:
    "User: What's our refund window?\nTool (lookup_policy): Refunds: 30 days from delivery.\nAssistant: 30 days from delivery.\nUser: And for electronics?",
  output: "Electronics can be returned within 90 days.",
});

console.log(result);
// { score: 1, label: "hallucinated", explanation: "..." }

Using Input Mapping

When your data has different field names or requires transformation, use input mapping. This is especially useful when you need to assemble a readable conversation from a list of messages.

Python
TypeScript

from phoenix.evals import LLM
from phoenix.evals.metrics import HallucinationEvaluator

llm = LLM(provider="openai", model="gpt-4o")
hallucination_eval = HallucinationEvaluator(llm=llm)

# Example with a list of messages and a separate response
eval_input = {
    "messages": [
        {"role": "user", "content": "What's our refund window?"},
        {"role": "tool", "content": "Refunds: 30 days from delivery."},
        {"role": "assistant", "content": "30 days from delivery."},
        {"role": "user", "content": "And for electronics?"},
    ],
    "response": "Electronics can be returned within 90 days.",
}

# Use input mapping with a lambda to render the conversation as a transcript
input_mapping = {
    "input": lambda x: "\n".join(
        f"{m['role'].capitalize()}: {m['content']}" for m in x["messages"]
    ),
    "output": "response",
}

scores = hallucination_eval.evaluate(eval_input, input_mapping)

For more details on input mapping options, see Input Mapping.

import { bindEvaluator, createHallucinationEvaluator } from "@arizeai/phoenix-evals";
import { openai } from "@ai-sdk/openai";

const hallucinationEvaluator = createHallucinationEvaluator({
  model: openai("gpt-4o"),
});

// Bind with input mapping for different field names
const boundEvaluator = bindEvaluator(hallucinationEvaluator, {
  inputMapping: {
    input: (data) =>
      data.messages
        .map((m) => `${m.role[0].toUpperCase()}${m.role.slice(1)}: ${m.content}`)
        .join("\n"),
    output: "response",
  },
});

const result = await boundEvaluator.evaluate({
  messages: [
    { role: "user", content: "What's our refund window?" },
    { role: "tool", content: "Refunds: 30 days from delivery." },
    { role: "assistant", content: "30 days from delivery." },
    { role: "user", content: "And for electronics?" },
  ],
  response: "Electronics can be returned within 90 days.",
});

For more details on input mapping options, see Input Mapping.

Configuration

For LLM client configuration options, see Configuring the LLM.

Viewing and Modifying the Prompt

You can view the latest versions of our prompt templates on GitHub. The evaluators are designed to work well in a variety of contexts, but we highly recommend modifying the prompt to be more specific to your use case. Feel free to adapt them.

Python
TypeScript

from phoenix.evals.metrics import HallucinationEvaluator
from phoenix.evals import LLM, ClassificationEvaluator

llm = LLM(provider="openai", model="gpt-4o")
evaluator = HallucinationEvaluator(llm=llm)

# View the prompt template
print(evaluator.prompt_template)

# Create a custom evaluator based on the built-in template
custom_evaluator = ClassificationEvaluator(
    name="hallucination",
    prompt_template=evaluator.prompt_template,  # Modify as needed
    llm=llm,
    choices={"hallucinated": 1.0, "grounded": 0.0},
    direction="minimize",
)

import { HALLUCINATION_CLASSIFICATION_EVALUATOR_CONFIG, createHallucinationEvaluator } from "@arizeai/phoenix-evals";
import { openai } from "@ai-sdk/openai";

// View the prompt template
console.log(HALLUCINATION_CLASSIFICATION_EVALUATOR_CONFIG.template);

// Create a custom evaluator with a modified template
const customEvaluator = createHallucinationEvaluator({
  model: openai("gpt-4o"),
  promptTemplate: HALLUCINATION_CLASSIFICATION_EVALUATOR_CONFIG.template, // Modify as needed
});

Using with Phoenix

Evaluating Traces

Run evaluations on traces collected in Phoenix and log results as annotations:

Running Experiments

Use the Hallucination evaluator in Phoenix experiments:

Using Evaluators in Experiments

API Reference

Python: HallucinationEvaluator
TypeScript: createHallucinationEvaluator

Faithfulness Evaluator - Grounds a single response in a single retrieved context
User Friction Evaluator - Detects friction expressed across a conversation

Get Started

Tracing

Evaluation

Datasets & Experiments

Prompts

Settings

Concepts

Resources

Overview

When to Use

Supported Levels

Input Requirements

Formatting the input

Output Interpretation

Usage Examples

Using Input Mapping

Configuration

Viewing and Modifying the Prompt

Using with Phoenix

Evaluating Traces

Running Experiments

API Reference

​Overview

​When to Use

​Supported Levels

​Input Requirements

​Formatting the input

​Output Interpretation

​Usage Examples

​Using Input Mapping

​Configuration

​Viewing and Modifying the Prompt

​Using with Phoenix

​Evaluating Traces

​Running Experiments

​API Reference

​Related

Overview

When to Use

Supported Levels

Input Requirements

Formatting the input

Output Interpretation

Usage Examples

Using Input Mapping

Configuration

Viewing and Modifying the Prompt

Using with Phoenix

Evaluating Traces

Running Experiments

API Reference

Related