> ## Documentation Index
> Fetch the complete documentation index at: https://arizeai-433a7140.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Completeness

> Assess whether every active user request in a conversation was actually completed.

## Overview

The **Completeness** evaluator classifies whether an assistant completed every
active request the user made over the course of a conversation. It measures
**finished work** — a delivered answer, a delivered artifact (including every
required component), or an action whose success is visible in the record — not
whether a request was merely acknowledged.

A conversation can finish its main task and still be incomplete when a
secondary request is dropped. For example, if the user asks the assistant to
reset a password **and** update a billing address, and the assistant only
resets the password, the conversation is incomplete.

<Info>
  Completeness does not judge correctness, quality, or whether completion was
  appropriate. A delivered answer can still be complete if it is factually wrong.
  A refusal, a clarifying question, or a report of a blocker is **not**
  completion. Withdrawn requests are listed but excluded from the decision.
</Info>

## Supported Levels

| Level       | Supported | Notes                                                                               |
| ----------- | --------- | ----------------------------------------------------------------------------------- |
| **Span**    | Yes       | Apply when a span contains the full conversation, including tool calls and results. |
| **Trace**   | Yes       | Useful when each trace is one conversation or agent run.                            |
| **Session** | Yes       | Evaluate the whole session when intentions span multiple traces.                    |

**Relevant span kinds:** AGENT, CHAIN, and LLM spans that preserve multi-turn
conversation history and tool activity.

## Input Requirements

| Field          | Type     | Description                                                                       |
| -------------- | -------- | --------------------------------------------------------------------------------- |
| `conversation` | `string` | Full conversation record to judge, including turns, tool calls, and tool results. |

Include early-turn requests. For agent traces, include tool calls and tool
results in `conversation` so action success can be verified. If tools are
omitted, the judge falls back to the visible dialogue.

## Output Interpretation

| Property      | Value                          | Description                                             |
| ------------- | ------------------------------ | ------------------------------------------------------- |
| `label`       | `"complete"` or `"incomplete"` | Classification result                                   |
| `score`       | `1.0` or `0.0`                 | `1.0` means every non-withdrawn intention was completed |
| `explanation` | `string`                       | Judge reasoning, including an `INTENTIONS:` list        |
| `direction`   | `"maximize"`                   | Higher aggregate scores are better                      |

## Usage Examples

<Tabs>
  <Tab title="Python" icon="python">
    ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    from phoenix.evals import LLM
    from phoenix.evals.metrics import CompletenessEvaluator

    evaluator = CompletenessEvaluator(
        llm=LLM(provider="openai", model="gpt-4o-mini"),
        temperature=0.0,
    )

    scores = evaluator.evaluate({
        "conversation": (
            "User: Reset my password and update the billing address.\n"
            "Assistant: Your password has been reset."
        ),
    })

    print(scores[0])
    # Score(name='completeness', score=0.0, label='incomplete', ...)
    ```
  </Tab>

  <Tab title="TypeScript" icon="js">
    ```typescript theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    import { createCompletenessEvaluator } from "@arizeai/phoenix-evals";
    import { openai } from "@ai-sdk/openai";

    const evaluator = createCompletenessEvaluator({
      model: openai("gpt-4o-mini"),
    });

    const result = await evaluator.evaluate({
      conversation:
        "User: Reset my password and update the billing address.\nAssistant: Your password has been reset.",
    });

    console.log(result);
    // { score: 0, label: "incomplete", explanation: "..." }
    ```
  </Tab>
</Tabs>

## Using Input Mapping

Map your trace or dataset fields into the evaluator's `conversation` field.

```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
input_mapping = {
    "conversation": lambda row: render_messages(row["messages"]),
}

scores = evaluator.evaluate(dataset_row, input_mapping)
```

See [Input Mapping](/docs/phoenix/evaluation/concepts-evals/input-mapping) for
additional mapping options.

## Viewing and Modifying the Prompt

The default prompt is maintained in the
[classification evaluator config](https://github.com/Arize-ai/phoenix/blob/main/prompts/classification_evaluator_configs/COMPLETENESS_CLASSIFICATION_EVALUATOR_CONFIG.yaml).
Adapt it when your application has domain-specific notions of what counts as an
intention.

```typescript theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import { createCompletenessEvaluator } from "@arizeai/phoenix-evals";

const evaluator = createCompletenessEvaluator({
  model,
  promptTemplate: `Conversation: {{conversation}}
Did the assistant complete every active user intention?`,
  choices: { complete: 1, incomplete: 0 },
});
```

## Configuration

For model and provider options, see
[Configuring the LLM](/docs/phoenix/evaluation/how-to-evals/configuring-the-llm).
Judge model choice changes accuracy on this evaluator; see [Benchmarks](#benchmarks)
for a comparison of `gpt-4o-mini`, `gpt-5.6`, and `claude-sonnet-4-6`.

## Using with Phoenix

* [Evaluating Phoenix Traces](/docs/phoenix/tracing/how-to-tracing/feedback-and-annotations/evaluating-phoenix-traces)
* [Logging LLM Evaluations](/docs/phoenix/tracing/how-to-tracing/feedback-and-annotations/llm-evaluations)
* [Using Evaluators in Experiments](/docs/phoenix/datasets-and-experiments/how-to-experiments/using-evaluators)

## Benchmarks

The default prompt was scored on a 45-example categorized synthetic suite. Every
model used the same prompt. Runs were local with `PHOENIX_TEST_TRACKING=false`.
See
[completeness.eval.ts](https://github.com/Arize-ai/phoenix/blob/main/js/benchmarks/evals-benchmarks/src/completeness.eval.ts)
for the example set.

| Model               | n  | Accuracy | Macro precision | Macro recall | Macro F1 | Misses |
| ------------------- | -- | -------- | --------------- | ------------ | -------- | ------ |
| `gpt-4o-mini`       | 45 | 0.84     | 0.83            | 0.84         | 0.84     | 7      |
| `gpt-5.6`           | 45 | 1.00     | 1.00            | 1.00         | 1.00     | 0      |
| `claude-sonnet-4-6` | 45 | 0.91     | 0.94            | 0.88         | 0.90     | 4      |

On this suite, `gpt-5.6` matched gold on every example. The other models mainly
missed cases that mix completeness with correctness, tool evidence, or
multi-part asks:

* **Correctness vs completeness.** Gold treats a delivered answer as complete even
  when it is wrong. `claude-sonnet-4-6` missed on these cases.
* **Claims without tool evidence.** Gold treats the visible reply as sufficient
  when no matching tool record exists. `gpt-4o-mini` and `claude-sonnet-4-6` missed here.
* **Partial delivery.** `gpt-4o-mini` labeled several multi-part and
  partial-tool cases complete.

<Accordion title="Disagreements (10 of 45)">
  Predicted labels where at least one model missed gold or the models disagreed
  with each other. **Bold** means the judge disagreed with gold.

  | Case                                    | Gold         | gpt-4o-mini      | gpt-5.6      | claude-sonnet-4-6 |
  | --------------------------------------- | ------------ | ---------------- | ------------ | ----------------- |
  | `[answered_despite_missing_context #0]` | `complete`   | `complete`       | `complete`   | **`incomplete`**  |
  | `[answered_despite_missing_context #1]` | `complete`   | `complete`       | `complete`   | **`incomplete`**  |
  | `[wrong_but_delivered #0]`              | `complete`   | `complete`       | `complete`   | **`incomplete`**  |
  | `[wrong_but_delivered #1]`              | `complete`   | **`incomplete`** | `complete`   | `complete`        |
  | `[withdrawn_intention #0]`              | `complete`   | **`incomplete`** | `complete`   | `complete`        |
  | `[claimed_but_not_done #1]`             | `incomplete` | **`complete`**   | `incomplete` | `incomplete`      |
  | `[claimed_without_tools #0]`            | `complete`   | **`incomplete`** | `complete`   | **`incomplete`**  |
  | `[multipart_cases #1]`                  | `incomplete` | **`complete`**   | `incomplete` | `incomplete`      |
  | `[multipart_cases #5]`                  | `incomplete` | **`complete`**   | `incomplete` | `incomplete`      |
  | `[tool_partial #0]`                     | `incomplete` | **`complete`**   | `incomplete` | `incomplete`      |
</Accordion>

## API Reference

* **Python:** [CompletenessEvaluator](https://arize-phoenix.readthedocs.io/projects/evals/en/latest/api/evals.html#module-phoenix.evals.metrics)
* **TypeScript:** [createCompletenessEvaluator](https://arize-ai.github.io/phoenix/modules/_arizeai_phoenix-evals.llm.html)

## Related

* [Correctness Evaluator](/docs/phoenix/evaluation/pre-built-metrics/correctness)
* [User Friction Evaluator](/docs/phoenix/evaluation/pre-built-metrics/user-friction)
* [Hallucination Evaluator](/docs/phoenix/evaluation/pre-built-metrics/hallucination)