Completeness

Overview

The Completeness evaluator classifies whether an assistant completed every active request the user made over the course of a conversation. It measures finished work — a delivered answer, a delivered artifact (including every required component), or an action whose success is visible in the record — not whether a request was merely acknowledged. A conversation can finish its main task and still be incomplete when a secondary request is dropped. For example, if the user asks the assistant to reset a password and update a billing address, and the assistant only resets the password, the conversation is incomplete.

Completeness does not judge correctness, quality, or whether completion was appropriate. A delivered answer can still be complete if it is factually wrong. A refusal, a clarifying question, or a report of a blocker is not completion. Withdrawn requests are listed but excluded from the decision.

Supported Levels

Level	Supported	Notes
Span	Yes	Apply when a span contains the full conversation, including tool calls and results.
Trace	Yes	Useful when each trace is one conversation or agent run.
Session	Yes	Evaluate the whole session when intentions span multiple traces.

Relevant span kinds: AGENT, CHAIN, and LLM spans that preserve multi-turn conversation history and tool activity.

Input Requirements

Field	Type	Description
`conversation`	`string`	Full conversation record to judge, including turns, tool calls, and tool results.

Include early-turn requests. For agent traces, include tool calls and tool results in conversation so action success can be verified. If tools are omitted, the judge falls back to the visible dialogue.

Output Interpretation

Property	Value	Description
`label`	`"complete"` or `"incomplete"`	Classification result
`score`	`1.0` or `0.0`	`1.0` means every non-withdrawn intention was completed
`explanation`	`string`	Judge reasoning, including an `INTENTIONS:` list
`direction`	`"maximize"`	Higher aggregate scores are better

Usage Examples

Python
TypeScript

from phoenix.evals import LLM
from phoenix.evals.metrics import CompletenessEvaluator

evaluator = CompletenessEvaluator(
    llm=LLM(provider="openai", model="gpt-4o-mini"),
    temperature=0.0,
)

scores = evaluator.evaluate({
    "conversation": (
        "User: Reset my password and update the billing address.\n"
        "Assistant: Your password has been reset."
    ),
})

print(scores[0])
# Score(name='completeness', score=0.0, label='incomplete', ...)

import { createCompletenessEvaluator } from "@arizeai/phoenix-evals";
import { openai } from "@ai-sdk/openai";

const evaluator = createCompletenessEvaluator({
  model: openai("gpt-4o-mini"),
});

const result = await evaluator.evaluate({
  conversation:
    "User: Reset my password and update the billing address.\nAssistant: Your password has been reset.",
});

console.log(result);
// { score: 0, label: "incomplete", explanation: "..." }

Using Input Mapping

Map your trace or dataset fields into the evaluator’s conversation field.

input_mapping = {
    "conversation": lambda row: render_messages(row["messages"]),
}

scores = evaluator.evaluate(dataset_row, input_mapping)

See Input Mapping for additional mapping options.

Viewing and Modifying the Prompt

The default prompt is maintained in the classification evaluator config. Adapt it when your application has domain-specific notions of what counts as an intention.

import { createCompletenessEvaluator } from "@arizeai/phoenix-evals";

const evaluator = createCompletenessEvaluator({
  model,
  promptTemplate: `Conversation: {{conversation}}
Did the assistant complete every active user intention?`,
  choices: { complete: 1, incomplete: 0 },
});

Configuration

For model and provider options, see Configuring the LLM. Judge model choice changes accuracy on this evaluator; see Benchmarks for a comparison of gpt-4o-mini, gpt-5.6, and claude-sonnet-4-6.

Using with Phoenix

Benchmarks

The default prompt was scored on a 45-example categorized synthetic suite. Every model used the same prompt. Runs were local with PHOENIX_TEST_TRACKING=false. See completeness.eval.ts for the example set.

Model	n	Accuracy	Macro precision	Macro recall	Macro F1	Misses
`gpt-4o-mini`	45	0.84	0.83	0.84	0.84	7
`gpt-5.6`	45	1.00	1.00	1.00	1.00	0
`claude-sonnet-4-6`	45	0.91	0.94	0.88	0.90	4

On this suite, gpt-5.6 matched gold on every example. The other models mainly missed cases that mix completeness with correctness, tool evidence, or multi-part asks:

Correctness vs completeness. Gold treats a delivered answer as complete even when it is wrong. claude-sonnet-4-6 missed on these cases.
Claims without tool evidence. Gold treats the visible reply as sufficient when no matching tool record exists. gpt-4o-mini and claude-sonnet-4-6 missed here.
Partial delivery. gpt-4o-mini labeled several multi-part and partial-tool cases complete.

Disagreements (10 of 45)

Predicted labels where at least one model missed gold or the models disagreed with each other. Bold means the judge disagreed with gold.

Case	Gold	gpt-4o-mini	gpt-5.6	claude-sonnet-4-6
`[answered_despite_missing_context #0]`	`complete`	`complete`	`complete`	`incomplete`
`[answered_despite_missing_context #1]`	`complete`	`complete`	`complete`	`incomplete`
`[wrong_but_delivered #0]`	`complete`	`complete`	`complete`	`incomplete`
`[wrong_but_delivered #1]`	`complete`	`incomplete`	`complete`	`complete`
`[withdrawn_intention #0]`	`complete`	`incomplete`	`complete`	`complete`
`[claimed_but_not_done #1]`	`incomplete`	`complete`	`incomplete`	`incomplete`
`[claimed_without_tools #0]`	`complete`	`incomplete`	`complete`	`incomplete`
`[multipart_cases #1]`	`incomplete`	`complete`	`incomplete`	`incomplete`
`[multipart_cases #5]`	`incomplete`	`complete`	`incomplete`	`incomplete`
`[tool_partial #0]`	`incomplete`	`complete`	`incomplete`	`incomplete`

API Reference

Python: CompletenessEvaluator
TypeScript: createCompletenessEvaluator

Get Started

Tracing

Evaluation

Datasets & Experiments

Prompts

Settings

Concepts

Resources

Overview

Supported Levels

Input Requirements

Output Interpretation

Usage Examples

Using Input Mapping

Viewing and Modifying the Prompt

Configuration

Using with Phoenix

Benchmarks

API Reference

​Overview

​Supported Levels

​Input Requirements

​Output Interpretation

​Usage Examples

​Using Input Mapping

​Viewing and Modifying the Prompt

​Configuration

​Using with Phoenix

​Benchmarks

​API Reference

​Related

Overview

Supported Levels

Input Requirements

Output Interpretation

Usage Examples

Using Input Mapping

Viewing and Modifying the Prompt

Configuration

Using with Phoenix

Benchmarks

API Reference

Related