Trace-Level Evals

How to measure LLM application performance at the trace level.

A trace is a complete record of all operations (spans) that occur during a single execution or request in your LLM application. Trace-level evaluations provide a holistic view of:

• Whether the overall workflow or task was completed successfully

• The quality and correctness of the end-to-end process

• Aggregate metrics such as latency, errors, and evaluation labels for the entire trace

• The success of multi-step workflows (ex: Agentic reasoning, RAG pipelines)

Trace-Level Evaluations via UI

Trace-Level Evaluations via UI

To run evaluations at the trace level in the UI, set the evaluator scope to “Trace” for each evaluator you want to operate at that level.

You can also apply filters to focus the evaluation on specific parts of a trace. If no filters are applied, the evaluation will consider the entire trace by default.

Trace-Level Evaluations via Code

Trace-Level Evaluations via Code

This step-by-step guide shows how to evaluate existing traces in code and log the results back to Arize.

1. Get trace data from Arize

from arize.exporter import ArizeExportClient
from arize.utils.types import Environments
from datetime import datetime, timedelta, timezone

client = ArizeExportClient()

primary_df = client.export_model_to_df(
    space_id = "YOUR_ARIZE_SPACE_ID",
    model_id = "YOUR_ARIZE_MODEL_ID",
    environment = Environments.TRACING,
    start_time = datetime.now(timezone.utc) - timedelta(days=7),
    end_time   = datetime.now(timezone.utc),
)

2. Prepare trace-level data for evaluation

Aggregate or filter your dataframe. For example, you might group by context.trace_id and concatenate relevant fields for your evaluation:

import pandas as pd

# Example: Aggregate all outputs in a trace
trace_df = (
    primary_df.groupby("context.trace_id")
      .agg({
          "attributes.input.value": "first",
          "attributes.output.value": lambda x: " ".join(x.dropna()),
          # Add other aggregations as needed
      })
)

3. Define your evaluation prompt

Create a prompt template for the LLM judge to assess the entire trace. For example, to judge overall correctness:

TRACE_EVAL_PROMPT = """
You are evaluating the overall quality and correctness of an LLM application's response to a user request.

You will be given:
1. The user input that initiated the trace
2. The full output(s) generated during the trace

##
User Input:
{attributes.input.value}

Trace Output:
{attributes.output.value}
##

Respond with exactly one word: `correct` or `incorrect`.
- `correct` → the trace achieves the intended goal.
- `incorrect` → the trace fails to achieve the goal or is low quality.
"""

4. Run the evaluation

Use Phoenix Evals to run your evaluation:

from phoenix.evals import create_classifier
from phoenix.evals.evaluators import async_evaluate_dataframe
from phoenix.evals.llm import LLM

llm = LLM(provider="openai", model="gpt-4o-mini")

correctness_evaluator = create_classifier(
    name="correctness",
    llm=llm,
    prompt_template=TRACE_EVAL_PROMPT,
    choices={"correct": 1.0, "incorrect": 0.0},
)

results_df = await async_evaluate_dataframe(
    dataframe=trace_df,
    evaluators=[correctness_evaluator],
)

5. Log the results back to Arize

Merge your evaluation results with original trace data to align trace_id with root span_id. Make sure to use the prefix trace_eval when naming columns before logging results:

from arize.pandas.logger import Client
from phoenix.evals.utils import to_annotation_dataframe
import ast

import pandas as pd
client = Client()

root_spans = primary_df[primary_df["parent_id"].isna()][["context.trace_id", "context.span_id"]]

# Merge results with root spans to align on trace_id
results_with_spans = pd.merge(
    results_df.reset_index(), root_spans, on="context.trace_id", how="left"
).set_index("context.span_id", drop=False)

# Format for logging
correctness_eval_df = to_annotation_dataframe(results_with_spans)

# using trace_eval prefix to rename columns
correctness_eval_df = correctness_eval_df.rename(columns={
    "label": "trace_eval.correctness.label",
    "score": "trace_eval.correctness.score",
    "explanation": "trace_eval.correctness.explanation"
})

client.log_evaluations_sync(correctness_eval_df, 'your-project-name')