How to build agent evals from traces

Here’s an overly simplistic truth: evals are tests for AI and traces are logs for AI. A trace records what an agent did across model calls, retrieval steps, tool calls, and other operations within one request. An eval applies a repeatable judgment to that behavior.

This tutorial connects the two. You will start with a small set of agent traces, identify recurring failures, write a deterministic code eval, build a focused LLM judge, validate that judge against human labels, and turn the resulting failures into a regression dataset.

For nuanced multi-step criteria, you will also learn when to use Agent-as-a-Judge, where an agentic harness reads trace data at runtime instead of relying on a fixed prompt and column mapping.

What you will build

We will use a simple financial research agent as the running example. It accepts a stock ticker and a focus area, then completes the task in two stages:

1. A research stage gathers current information.
2. A report-writing stage turns that research into an analysis.

The agent is intentionally imperfect. Across a small batch of traces, it sometimes drifts away from the requested ticker, makes claims that are difficult to verify, gives a summary without an actionable conclusion, or says that it saved a file even though no file exists.

By the end of the tutorial, you will have:

• a requirement card that defines successful behavior
• a failure taxonomy derived from traces
• a deterministic ticker-coverage eval
• a state-based artifact eval
• a faithfulness judge grounded in the research stage
• a decision rule for when to use Agent-as-a-Judge
• a custom actionability judge
• a human-labeled calibration set
• precision and recall measurements for the judge
• passing and failing datasets for experiments
• a path for running the same evals on production traffic

Where this fits, and what it deliberately skips

This guide goes deep on the implementation layer, but it’s part of a larger set. It assumes you have decided what to measure and are now writing the evaluator.

• For the conceptual model (evaluating outcome, path, decision, and reliability rather than one correct trajectory), read the agent-native evaluation framework.
• For instrumentation depth (what to trace, how to design spans, OpenTelemetry and OpenInference, and redaction before export), read AI agent tracing and evaluation.
• For the end-to-end production workflow, release gates, and team process, read the production agent evaluation workflow in the AI Agent Handbook.
• For the full LLM-as-a-judge primer, evaluator library, and research basis, read the LLM-as-a-judge guide.
• For metric selection by agent type, read agent evaluation metrics.
• For the business case for evals, read why you need evals for AI agents.
• For Agent-as-a-Judge product mechanics and current availability, read the Agent-as-a-Judge documentation.

Key terms used in this tutorial

Span and trace are core OpenTelemetry tracing concepts. Sessions are an application-level grouping represented by a session identifier on related telemetry; in Arize and OpenInference, related traces are grouped with session.id. If those terms are new, read the tracing and session concepts first.

1. Start with a working mental model

Traditional tests work well when a program should return one predictable result. Given a fixed input, you can assert an exact value, compare a data structure, or inspect a state change.

Agent outputs aren’t quite like that. Unlike traditional software, we can’t know exactly what an agent or LLM will do until runtime. Moreover, the same request can produce different wording, different plans, or different tool sequences while still completing the task correctly. That makes exact-string assertions too brittle for many behaviors.

Here’s a useful mental model:

• Traces tell you what happened. They record the execution.
• Evals tell you whether it was acceptable. They apply a criterion to the execution or its result.

The execution is usually represented as related spans with parent-child relationships. One span might represent a model call, another a web search, and another a tool invocation. The complete request is a trace. Several traces can belong to one multi-turn session.

Session: research conversation

Trace 1: research request

  Span: agent turn

    Span: web search

    Span: web search

    Span: model synthesis

Trace 2: report request

  Span: agent turn

    Span: model call

    Span: artifact write attempt

An evaluator should operate at the smallest scope that contains the evidence needed to make the decision. A retrieval relevance check may need one span. A task-completion check may need the full trace. A conversation-resolution check may need the entire session.

This is a narrower claim than the full evaluation framework. The framework article explains which dimensions of agent behavior deserve evaluation at all; the agent-native evaluation framework covers outcome, path, decision, and reliability, and argues for invariants rather than one reference trajectory. This tutorial takes those dimensions as given and shows how to implement one evaluator at a time.

2. Three successful runs are not a test suite

A common agent development loop looks reasonable at first:

1. Run the agent on a few prompts.
2. Read the outputs.
3. Decide that they look good.
4. Ship.

But the problem is coverage. Three runs tell you that the agent succeeded three times. They do not tell you how it behaves across ambiguous requests, sparse data, multiple entities, failed tools, long conversations, or repeated attempts.

A useful eval suite should answer two different questions.

What is a capability eval?

A capability eval asks whether the agent can perform a behavior that it does not yet handle consistently. Early in development, many examples should fail. Those failures define the next improvement target.

Here are some examples:

• Can the research agent separate historical performance from forward-looking analysis?
• Can the support agent recover when the first tool call times out?
• Can the coding agent fix a bug that spans several files?

A capability suite that passes at 100 percent may no longer be challenging enough to guide improvement.

What is a regression eval?

A regression eval protects behavior the agent already performs reliably. A confirmed production failure often becomes a regression case after the team fixes it.

Examples include:

• The report must cover the ticker the user requested.
• A refund above the automatic limit must require approval.
• A generated configuration file must parse.
• A support session must not end with an unresolved request labeled as resolved.

A capability eval can graduate into the regression suite once the behavior is stable. Keep the distinction visible in your test metadata so teams know which failures should block a release and which represent a longer-term improvement target.

id: report_has_actionable_conclusion

role: capability

release_behavior: track

---

id: requested_ticker_is_covered

role: regression

release_behavior: block

This is also where north-star metrics and guardrails separate. A north star measures an outcome you want to improve whereas a guardrail represents a condition the system cannot violate.

Once you are ready to attach those roles to an actual release gate (example-level thresholds, dataset-level pass rates, and which failures block a deploy), you can explore CI/CD for LLM applications and CI/CD for automated experiments in the docs.

3. Choose among code evals, LLM-as-a-Judge, Agent-as-a-Judge, and human review

Most production evaluation stacks use a mix of the following.

Use the least subjective method that can reliably answer the question.

Here’s a practical litmus test: could two reviewers, given the same span attributes and the same rubric, ever disagree on the answer? If the answer is no, the check is mechanical and belongs in code. If the answer is yes, choose an LLM-as-a-Judge, Agent-as-a-Judge, or human review based on how much context and investigation the decision requires.

Use code when the condition is observable

Good code-eval questions include:

• Did the output parse as JSON?
• Did the tool arguments satisfy the schema?
• Did the agent use a prohibited tool?
• Did the requested record change in the database?
• Did the generated code pass its tests?
• Does the artifact exist?
• Was latency below the required threshold?

Code evaluators avoid model-inference cost and are usually cheap enough to run broadly. Sample them only when data volume or computation makes full coverage unnecessarily expensive; reserve most cost-based sampling decisions for model- or agent-based judges. Run deterministic checks broadly and reserve sampling decisions for the judges.

Use an LLM judge when the condition depends on meaning

Good judge questions include:

• Is the report faithful to the supplied research?
• Does the answer address the user’s actual request?
• Is the recommendation actionable?
• Was the escalation appropriate?
• Did the agent choose a reasonable recovery strategy?
• Did the session resolve the user’s problem?

For the underlying method including how judges work, which prebuilt evaluators exist, how prompts are structured, and what the research says about judge reliability, read the LLM-as-a-judge guide. This tutorial covers only how to author one judge for one failure you observed. If you need a broader map of evaluation types before choosing, classes of LLM evaluations is the better starting point.

Use Agent-as-a-Judge when the evaluator needs to investigate the trace

A standard LLM-as-a-Judge works best when the evidence can be mapped into a stable prompt template. Agent-as-a-Judge is for criteria that require the evaluator to explore trace data at runtime, reason across several fields or spans, or inspect a multi-step trajectory before deciding what matters.

In Arize AX, Agent-as-a-Judge runs an agentic harness over exported trace data and scores from natural-language instructions. It does not require the fixed column mappings used by template-style LLM judges. Use it for trajectory quality, recovery behavior, or rubrics whose evidence is spread across a trace. Keep deterministic checks in code and use standard LLM judges for simpler high-volume checks. At publication time, Agent-as-a-Judge is in closed Enterprise beta and supports a Claude Code harness; check the Arize AX docs for current availability and supported harnesses.

Stack the layers instead of choosing one

A practical pattern is layered coverage rather than one evaluator for everything. Use code for mechanical invariants, standard LLM judges for stable semantic criteria, Agent-as-a-Judge for complex multi-step judgments that require runtime trace exploration, and human review for ambiguous or high-consequence cases.

Human review belongs in the loop when evidence is incomplete, evaluators disagree, a new failure pattern appears, or a wrong automated verdict could permit a high-impact action. In Arize AX, this is what human review and labeling queues exist for: routing a defined slice of traces to reviewers and capturing structured annotations rather than informal comments.

Think of these layers as overlapping controls. A schema check can catch malformed data. A standard LLM judge can catch a semantic gap. Agent-as-a-Judge can reason across a trajectory that is awkward to compress into one fixed template. A human can recognize an ambiguous policy case.

Each layer covers failures that another layer may miss.

4. Evaluate at span, trace, or session scope

Scope determines what evidence the evaluator can see.

Span-level example

A retrieval span contains the query and returned documents. That is enough evidence for a relevance judge.

Question: Did this retrieval return information relevant to the requested ticker?

Scope: retrieval span

Evidence: query + retrieved documents

Trace-level example

A report-writing trace contains the input, research context, tool activity, and final output. That is enough evidence for an end-to-end report-quality evaluator.

Question: Did the agent produce an actionable, grounded report for the requested company?

Scope: trace

Evidence: request + research + tool results + report

Session-level example

A banking assistant asks which account the user means in one turn, then provides the balance in the next. Evaluating either trace alone would miss whether the conversation resolved the request.

Question: Did the user receive the requested balance after clarification?

Scope: session

Evidence: all turns in the conversation

Choose scope from the evidence backward. Starting from the UI location or the span type often leads to an evaluator that cannot see enough context.

Two practical constraints before you pick a wider scope

Scope is not free, and the mechanics matter once you move above the span.

Results land in different places. In Arize AX, a span evaluator writes back to eval.{name}.label, .score, and .explanation. A trace evaluator writes trace_eval.{name}.* and a session evaluator writes session_eval.{name}.*. Dashboards, filters, and alerts that reference the span-level columns will not see trace- or session-level results, so decide the scope before you build the monitor on top of it.

Wider scope changes how evidence is assembled, but current online tasks do not require you to dump every span into one prompt. For trace and session evaluators, multi-span queries can select the span patterns that matter and map variables from named subqueries; session evals can also inject a formatted conversation built from root spans. Use the smallest sufficient scope, and use multi-span queries when span order or span roles matter. Agent-as-a-Judge is another option when the evaluator needs to explore trace data at runtime rather than rely on fixed mappings.

Session scope has a hard prerequisite: related spans must carry a non-empty session.id. Without it, Arize has nothing to group into a session. See session setup and trace/session evals for configuration.

5. Measure capability and consistency separately

A single success rate can hide an important distinction: an agent may be capable of solving a task while remaining too inconsistent for unsupervised use.

Three related metrics help:

• pass@1: Did one attempt succeed?
• pass@k: Across k attempts, did at least one succeed?
• pass^k: Across k attempts, did every attempt succeed?

Assuming each attempt succeeds independently with probability p, a 75 percent per-attempt success rate over ten attempts gives:

pass@10 = 1 - (1 - 0.75)^10 ≈ 99.9999%

pass^10 = 0.75^10 ≈ 5.6%

Both numbers describe the same agent. The first asks whether repeated attempts can eventually produce one acceptable result. The second asks whether the behavior stays acceptable every time. A human reviewing several candidate outputs may care about pass@k; an unattended agent that sends messages or changes external state should care much more about pass^k and critical-failure rate.

Real agent failures are correlated rather than independent, so treat these as intuition rather than an estimator. The point for this tutorial is narrower: decide which of the two your evaluator is measuring before you record a pass rate, because running each dataset example once measures pass@1 and nothing else. If consistency is the requirement, the eval harness has to run each example several times.

For how these metrics fit the wider measurement picture, including reliability under long multi-step workflows, see the agent-native evaluation framework. For choosing metrics by agent type, see agent evaluation metrics.

6. Capture the evidence your evals will need

You cannot build a useful trace-based eval if the trace omits the evidence that defines success or failure. Most evaluators in this tutorial depend on trace data, so instrumentation gaps become evaluation gaps. The state-based artifact check is the exception because it queries the environment directly.

At minimum, capture:

• the user or task input
• model inputs and outputs
• retrieved context
• tool names, arguments, results, and errors
• relevant state before and after actions
• prompt, model, and agent version
• latency, token use, and cost metadata
• a session identifier for multi-turn workflows

A minimal Arize AX setup for an OpenAI-based Python application looks like this:

pip install arize-otel openai openinference-instrumentation-openai

import atexit

import os

from arize.otel import register

from openinference.instrumentation.openai import OpenAIInstrumentor

# project_name is required. Without it, trace export fails.

# Set ARIZE_COLLECTOR_ENDPOINT for EU or Canada; the default targets the US cluster.

tracer_provider = register(

    space_id=os.environ["ARIZE_SPACE_ID"],

    api_key=os.environ["ARIZE_API_KEY"],

    project_name="financial-analyst-agent",

)

# Register and instrument before making model calls; follow integration-specific ordering requirements.

OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

# Spans are exported in batches. A script or notebook that exits before the

# batch flushes will silently drop the traces you were about to evaluate.

atexit.register(tracer_provider.shutdown)

Three details cause most of the “I ran the agent but there are no traces” reports:

1. project_name is required. Export fails without it.
2. Initialization order can matter. Register the tracer provider and instrument the provider or framework before making calls. Some integrations patch framework imports or classes and must be initialized before those objects are imported or constructed, so follow the integration-specific setup rather than relying on one universal ordering rule.
3. Short-lived processes must flush. The batch span processor exports asynchronously. A tutorial script that runs four inputs and exits will lose spans unless it calls force_flush() or shutdown().

Use the instrumentor that matches your provider or framework. Provider-SDK instrumentation captures model calls and the tool-call requests emitted by the model, but application-side tool execution may require a framework instrumentor or manual spans. If your agent runs its own tool loop against a raw provider SDK, explicitly trace the tool execution, result, errors, and turn boundaries that your evaluators need. Framework instrumentors often cover more of that execution path automatically.

That distinction, span design, multi-agent trace structure, and redaction before export are all covered in depth in AI agent tracing and evaluation. For the current package list and framework-specific setup, follow instrument your agent and manual instrumentation in the docs. Prompts, tool arguments, and retrieved records frequently contain personal or regulated data, so apply masking and redaction before export rather than after.

Then run a deliberately varied input set. A small tutorial set might include:

EVAL_INPUTS = [

    {

        "tickers": ["AMZN"],

        "focus": "AWS growth, margins, and valuation",

    },

    {

        "tickers": ["RIVN"],

        "focus": "cash runway, deliveries, and key risks",

    },

    {

        "tickers": ["MSFT", "GOOGL"],

        "focus": "compare AI capital spending and margin impact",

    },

    {

        "tickers": ["BRK.B"],

        "focus": "operating earnings and insurance exposure",

    },

]

The goal is variation versus volume for its own sake. Include common requests, difficult cases, sparse-data entities, multiple entities, and known edge cases.

7. Read traces before you automate judgment

This is the highest-value step in the workflow. It’s also the step teams skip over a lot, in our experience.

Before you write an evaluator, read a representative set of traces end to end. Look at the input, each retrieval, every tool call, intermediate outputs, the final response, and any resulting state change.

Automation written before error analysis tends to measure what is easy to count. By contrast, trace review tells you what actually matters.

Write down what success means first

You cannot evaluate an agent against an undefined standard. To that end, you should turn your product requirements into observable conditions.

For the financial research agent:

agent: financial_research_agent

successful_behavior:

  outcome:

    - covers every requested ticker

    - produces a report for the requested focus area

  evidence:

    - factual claims are supported by captured research

    - recent figures can be traced to a source in the research turn

  usefulness:

    - distinguishes historical summary from forward-looking analysis

    - provides a concrete recommendation or next step when the user asks for one

  state:

    - never claims that an artifact was saved unless the artifact exists

  boundaries:

    - does not answer outside the requested financial research task

Each line should point to evidence that exists in the trace or environment. If the evaluator cannot observe the evidence, either improve instrumentation or route the case to review. This card is the shortest useful version of an evaluation specification; the agent-native evaluation framework covers the fuller specification format.

What is open coding, and why start there?

Open coding comes from qualitative research. Read each trace and record the failure in plain language without forcing it into a predefined taxonomy.

Examples include:

• Report covered AWS but never discussed Amazon as a whole.
• Report quoted a price that could not be traced to research.
• Report discussed risks but gave no recommendation.
• Final response claimed that rivian-report.md was saved, but no file existed.
• Agent repeated the same search with slightly different wording.
• Agent used correct research but drew an unsupported conclusion.

Stay concrete. “Bad answer” is too vague to become an evaluator. A note is specific enough when a second reviewer could find the same defect in the same trace without asking you what you meant.

What is axial coding, and how does it produce a taxonomy?

After reading the traces, group related observations into categories.

This order matters. Predefined categories can make you overlook unexpected failures. Open coding lets the evidence lead. Axial coding turns the evidence into a taxonomy you can automate.

The output of this step is a taxonomy specific to your agent. If you want a worked reference taxonomy to compare against (ie, failure families mapped to the span, trace, or session where each becomes observable), AI agent tracing and evaluation contains one for a support agent. Use it to check your own coverage, not as a substitute for reading your traces.

Prioritize by frequency and severity

A practical prioritization rule is:

priority ≈ frequency × severity

A frequent wording issue may matter less than a rare unauthorized action. Add a third factor, detectability, when the failure is difficult for users or operators to notice.

priority ≈ frequency × severity × difficulty_of_detection

Use this ranking to choose the first eval. Start with a failure that matters, appears often enough to measure, and has evidence you can inspect.

Diagnose the cause instead of just diagnosing the symptom

The same bad final answer can come from different parts of the system. This table is the reason trace-level evidence beats output-level evidence: each row looks identical if all you have is the final response.

Without this distinction, teams can spend a week changing the prompt for a retrieval problem.

8. Build a deterministic code eval

Start with the simplest objective condition that catches a real failure.

For the financial agent, one requirement is the report should cover every requested ticker. Because the input already contains structured ticker symbols, the evaluator should use that field rather than trying to infer tickers from natural language.

from __future__ import annotations

import re

from dataclasses import dataclass

from typing import Literal, Sequence

Label = Literal["pass", "fail", "not_applicable"]

@dataclass(frozen=True)

class EvalResult:

    label: Label

    score: float | None

    explanation: str

def ticker_coverage_eval(

    expected_tickers: Sequence[str],

    report: str,

) -> EvalResult:

    """Check whether the report explicitly covers every requested ticker."""

    if not expected_tickers:

        return EvalResult(

            label="not_applicable",

            score=None,

            explanation="No expected tickers were supplied.",

        )

    missing: list[str] = []

    normalized_report = report.upper()

    for ticker in expected_tickers:

        token = re.escape(ticker.upper())

        pattern = rf"(?<![A-Z0-9]){token}(?![A-Z0-9])"

        if re.search(pattern, normalized_report) is None:

            missing.append(ticker)

    if missing:

        return EvalResult(

            label="fail",

            score=0.0,

            explanation=f"Report does not mention: {', '.join(missing)}.",

        )

    return EvalResult(

        label="pass",

        score=1.0,

        explanation="Report mentions every requested ticker.",

    )

This check is intentionally narrow. It doesn’t claim the analysis is correct or complete, but instead verifies one requirement with a deterministic result.

Two details are worth copying even if the rest of the logic changes. The boundary pattern prevents short tickers such as AI or CAT from matching inside larger alphanumeric strings, and naive substring matching is a common way a coverage eval quietly passes the wrong text. The not_applicable label exists so that inapplicable examples do not silently count as passes; exclude it from the denominator when you compute a pass rate, or your reported score will drift upward as the dataset grows.

Run the same logic as a platform evaluator

The function above is a pure function so that you can unit-test it and run it anywhere. To run the same check continuously on production traces, Arize AX expects a small class whose evaluate method reads the span and returns an EvaluationResult:

# Current custom CodeEvaluator shape

from typing import Any, Mapping, Optional

from arize.experimental.datasets.experiments.evaluators.base import (

    CodeEvaluator,

    EvaluationResult,

    JSONSerializable,

)

class TickerCoverageEvaluator(CodeEvaluator):

    def evaluate(

        self,

        *,

        report: Optional[str] = None,

        dataset_row: Optional[Mapping[str, JSONSerializable]] = None,

        **kwargs: Any,

    ) -> EvaluationResult:

        row = dataset_row or {}

        raw_tickers = row.get("attributes.metadata.tickers") or []

        expected = (

            [str(x) for x in raw_tickers]

            if isinstance(raw_tickers, list)

            else [str(raw_tickers)]

        )

        result = ticker_coverage_eval(expected, report or "")

        return EvaluationResult(

            label=result.label,

            score=result.score,

            explanation=result.explanation,

        )

In the task UI, map report to the output field (for example attributes.output.value) and add attributes.metadata.tickers as an Additional Span Attribute so it is available through dataset_row. The platform handles task triggering, filtering, sampling, and writing eval results back to the trace. Keep the requested tickers as structured metadata instead of parsing them back out of prose.

Check state instead of trusting the transcript

An agent’s confirmation message is not evidence that an action occurred. When the task changes external state, inspect that state directly.

The example agent occasionally reported that a Markdown report had been saved even though its sandbox did not permit file writes. A state-based eval catches the mismatch.

from pathlib import Path

def artifact_exists_eval(expected_path: Path) -> EvalResult:

    """Verify that a required artifact exists and contains data."""

    if not expected_path.exists():

        return EvalResult(

            label="fail",

            score=0.0,

            explanation=f"Expected artifact does not exist: {expected_path}",

        )

    if not expected_path.is_file():

        return EvalResult(

            label="fail",

            score=0.0,

            explanation=f"Expected artifact is not a file: {expected_path}",

        )

    if expected_path.stat().st_size == 0:

        return EvalResult(

            label="fail",

            score=0.0,

            explanation=f"Expected artifact is empty: {expected_path}",

        )

    return EvalResult(

        label="pass",

        score=1.0,

        explanation=f"Artifact exists and is non-empty: {expected_path}",

    )

The same principle applies to other agents:

• Refund agent: query the refund and case records.
• Coding agent: run the test suite and inspect the repository diff.
• CRM agent: verify the intended record and fields.
• Scheduling agent: inspect the calendar event and attendees.
• Data agent: validate the generated table or query result.

Rule: Grade the resulting state whenever the state is observable.

One caveat on where this eval runs. A state check needs access to the environment the agent acted on, so it belongs in your test harness or in a step that runs close to the agent, not in a platform evaluator reading a span days later. What the trace can carry is the result of that check: write the observed state into a span attribute at the time of the action, and a platform evaluator can then compare the claim against the recorded state.

Return evidence with the verdict

A useful evaluator returns a label, a score when needed, and an explanation that identifies the failed condition. The explanation should make the failure easy to inspect rather than merely repeat the label. “Report does not mention: RIVN” sends a developer to the defect; “actionability check failed” sends them back to the trace to guess.

9. Build a focused LLM-as-a-Judge evaluator

Code can verify that the ticker appears. It cannot decide whether the report turns research into useful analysis.

That is a semantic question, so we need an LLM judge.

Choose the right semantic target

A common mistake is selecting a broad evaluator whose evidence does not match the task.

Consider two different questions:

• Correctness: Is this claim true in the world?
• Faithfulness: Is this claim supported by the context supplied to the agent?

A correctness judge needs an authoritative reference or reliable access to current facts. A faithfulness judge only needs the research context and final report. Choosing between them is not a stylistic preference: a correctness judge without a reference will fall back on the judge model’s training data, which is exactly the unreliable narrator you were trying to measure.

For our two-stage workflow, the evidence may live across several spans in one trace or across related traces in a session, depending on how you instrument the boundary. The evaluator needs three pieces of evidence:

user_request     <- the request that opened the trace

research_context <- output of the research 

report           <- output of the report-writing

In Arize AX, a template-style LLM-as-a-Judge connects those inputs through column mappings: each {variable} in the template is mapped to a field path such as attributes.input.value or attributes.output.value. Keeping the template variables abstract and doing the binding in the mapping lets the same judge definition be reused across tasks with different data sources. Agent-as-a-Judge uses a different path: its harness reads trace data at runtime and does not require fixed column mappings.

This lets the judge ask whether the report stays grounded in the research the agent actually used. It does not ask the judge to reconstruct the current financial world from its own training data.

Choosing the right evaluator often matters more than tuning a judge that cannot see the required evidence.

Write one evaluator per dimension

Avoid a single prompt that tries to judge correctness, tone, completeness, policy compliance, actionability, and formatting at once. A failed verdict would not tell you what to fix.

Separate the dimensions:

faithfulness_eval

subject_coverage_eval

actionability_eval

forward_looking_analysis_eval

artifact_state_eval

Each evaluator should have one target, one evidence contract, and one set of labels.

Build the rubric from observed failures

The reports sometimes discussed future conditions but stopped before telling the reader what to do. That observation (from error analysis, not from a metric catalog) becomes an actionability rubric.

You are reviewing a report produced by a financial research agent.

Evaluation target

Determine whether the report gives the user an actionable conclusion for the

request they made.

Evidence you may use

Use only the user request, the research context, and the final report. Do not

introduce outside facts.

PASS when

- The report gives a clear recommendation, decision, or next step that matches

  the user's request.

- The recommendation is connected to specific evidence in the report.

- The report names at least one concrete driver or risk that could change the

  recommendation.

FAIL when

- The report only summarizes facts or historical performance.

- The report lists opportunities and risks without a conclusion.

- The recommendation is generic, such as "consider several factors."

- The conclusion is unsupported by the supplied research context.

INSUFFICIENT_EVIDENCE when

- The user request, research context, or report is missing enough information

  to apply the criteria.

<user_request>

{user_request}

</user_request>

<research_context>

{research_context}

</research_context>

<report>

{report}

</report>

Configure the possible labels separately in the evaluator system rather than asking the model to invent a scale:

PASS -> score 1

FAIL -> score 0

INSUFFICIENT_EVIDENCE -> no score; route for review

A binary choice works well when the evidence is complete and the decision is discrete. Add an abstention label when missing context would otherwise force the judge to guess.

Use the lowest-variance model settings the provider supports. Set temperature to 0 when the selected model exposes that control, but do not treat temperature 0 as a determinism guarantee. Repeatability still has to be measured on the calibration set, especially when the judge model or provider changes.

Give the judge domain and task context

Generic framing such as “you are an expert evaluator” does little to define the actual decision. Provide the system context, the quality dimension, the evidence it may use, and the decision the judge must make.

The useful context here is that the system produces financial research reports and the evaluator measures actionability. That information narrows the task without asking the judge to perform a vague expert role.

Include labeled examples

Examples turn abstract criteria into concrete boundaries.

Example: PASS

User request:

Assess ACME's cash runway and tell me what an investor should watch next.

Report excerpt:

Hold. At the current quarterly burn, reported liquidity supports roughly seven

quarters. The next decision point is the Q3 delivery update. A material miss on

deliveries or a higher burn rate would weaken the runway and change the hold

case.

Why it passes:

The report gives a decision, connects it to evidence, and names a measurable

condition that could change the recommendation.

Example: FAIL

User request:

Assess ACME's cash runway and tell me what an investor should watch next.

Report excerpt:

The company has several opportunities and risks. Investors should monitor

market conditions and consider a variety of factors before making a decision.

Why it fails:

The report contains no specific conclusion, evidence-linked next step, or

observable decision trigger.

Use several examples across the difficult boundary cases. Highly repetitive examples can cause the judge to copy superficial patterns, so vary the wording and scenario. Take the examples from your held-out iteration split, not from the validation split you plan to score against.

Ask for a concise evidence-based explanation

The explanation should name the criterion and point to observable evidence. It should not be treated as proof that the verdict is correct. A judge can produce a fluent explanation for a wrong label, which is why section 10 exists.

A useful result looks like this:

{

  "label": "FAIL",

  "score": 0,

  "explanation": "The report describes future risks but gives no recommendation or decision trigger for the user."

}

Use pairwise judging for genuinely open-ended work

For creative, research, or synthesis tasks where an absolute score is difficult to anchor, compare two outputs against the same criteria.

Ask:

Which response better satisfies the user's request and the evaluation criteria?

Run the comparison in both orders:

1. A versus B
2. B versus A

If the winner changes after the order changes, treat the result as a tie or inconclusive. This reduces the effect of position bias. When one judge is not enough for a high-stakes comparison, aggregating several judges is the next step; see LLM-as-a-jury.

Agent-as-a-Judge is the next step when a fixed prompt and column mapping are too rigid. The harness reads exported trace data at runtime and scores from natural-language instructions, which is useful for trajectory quality, recovery behavior, and other multi-field judgments. Keep the focused LLM judge for stable, high-throughput checks where the relevant evidence is known in advance.

10. Evaluate the evaluator

You should treat the judge like a classifier for a categorical evaluator like the one in this tutorial.

The judge receives an example and predicts a label. But you need a human-reviewed reference set to measure whether those predictions match the standard your team intends to enforce.

Create a calibration set

1. Select representative traces, including edge cases.
2. Give reviewers the same rubric used by the judge.
3. Label each example.
4. Record disagreements and adjudicate ambiguous cases.
5. Hold back a validation split while tuning the prompt.

A 75/25 iteration-versus-holdout split is a workable starting point when the set is large enough. With smaller datasets, prioritize coverage across important failure slices and keep a truly untouched holdout. A handful of examples is useful for debugging the prompt, but it is too small for a stable metric. There is no universal minimum sample size: 50 to 100 labeled examples can be a useful starting point for prompt iteration, but release-gate confidence should be based on failure prevalence, slice coverage, and the false-positive and false-negative rates the downstream action can tolerate.

Two operational notes:

• Reviewers need a fixed label schema, not a comment box, or you cannot compute agreement afterward; in Arize AX, annotation configs define that schema and labeling queues route the traces to reviewers.
• A calibration set drawn only from failures will overstate recall and understate precision, because the base rate is wrong. Instead, you should sample the slice you actually intend to run the judge on.

The product-specific version of this loop, including how to feed human labels back into judge iteration, is documented in align evals to human feedback and covered in more depth in measuring human and LLM judge alignment.

Define the positive class explicitly

For a failure detector, define FAIL as the positive class. That makes the metrics easier to interpret:

• Precision: When the judge says FAIL, how often does the human label also say FAIL?
• Recall: Of all human-labeled failures, how many did the judge catch?

from __future__ import annotations

from dataclasses import dataclass

from typing import Iterable

ABSTAIN_LABELS = {"INSUFFICIENT_EVIDENCE", "NEEDS_REVIEW"}

@dataclass(frozen=True)

class BinaryMetrics:

    precision: float

    recall: float

    coverage: float

    abstentions: int

    true_positives: int

    false_positives: int

    false_negatives: int

    true_negatives: int

def failure_detection_metrics(

    human_labels: Iterable[str],

    judge_labels: Iterable[str],

) -> BinaryMetrics:

    """Compare decided judge labels against human labels."""

    pairs = list(zip(human_labels, judge_labels, strict=True))

    decided = [(h, j) for h, j in pairs if j not in ABSTAIN_LABELS]

    abstentions = len(pairs) - len(decided)

    tp = sum(h == "FAIL" and j == "FAIL" for h, j in decided)

    fp = sum(h != "FAIL" and j == "FAIL" for h, j in decided)

    fn = sum(h == "FAIL" and j != "FAIL" for h, j in decided)

    tn = sum(h != "FAIL" and j != "FAIL" for h, j in decided)

    precision = tp / (tp + fp) if tp + fp else 0.0

    recall = tp / (tp + fn) if tp + fn else 0.0

    coverage = len(decided) / len(pairs) if pairs else 0.0

    return BinaryMetrics(

        precision=precision,

        recall=recall,

        coverage=coverage,

        abstentions=abstentions,

        true_positives=tp,

        false_positives=fp,

        false_negatives=fn,

        true_negatives=tn,

    )

strict=True makes a length mismatch between the two label lists raise instead of silently truncating. Report the raw counts alongside the rates, and report coverage when the judge can abstain. An abstention is neither an ordinary negative nor a detected failure; if abstentions route to human review, evaluate that routing workflow separately.

Choose the tradeoff based on the action tied to the evaluator.

Study disagreements first

Disagreements are the most informative examples in the calibration set.

For each disagreement, ask:

• Did the judge receive the same evidence as the human?
• Is the rubric ambiguous?
• Are two criteria mixed together?
• Did the human label drift from the rubric?
• Does the example belong to a missing category?
• Is the judge rewarding length, confidence, or style instead of the target behavior?

The actionability rubric in this tutorial originally accepted “forward-looking analysis.” Review showed that a report could discuss the future while avoiding a recommendation. Tightening the criterion to “forward-looking analysis with a specific recommendation” better matched the intended standard. Note which artifact changed: the fix was to the rubric, not to the judge model or the agent.

Watch for predictable judge biases

Common failure patterns include:

• Position bias: preferring the first or second response in pairwise comparison
• Length bias: rewarding longer responses even when the extra text adds little
• Confidence bias: accepting a fluent, certain answer with weak evidence
• Self-preference: rating outputs from the same model family more favorably

A cross-family judge may reduce correlated preferences, but calibration against human-reviewed examples is the test that matters.

Apply the fairness test

Open failed examples and ask whether the failure seems fair.

A good failed result makes the defect and the failed criterion understandable. If reasonable reviewers repeatedly open a failed trace and conclude that the output is acceptable, fix the evaluator before optimizing the agent against it. Optimizing an agent against a miscalibrated judge is the most expensive mistake in this workflow, because the metric improves while the product does not.

Rerun this validation whenever the rubric or scoring instructions, the judge model or harness, the data mappings, or the agent’s behavior changes. A judge validated six months ago on a different agent version is an unvalidated judge.

11. Turn passing and failing traces into datasets

Once an evaluator finds a real failure, make sure you preserve it.

A failure dataset answers whether the candidate change fixes known bad behavior. A passing dataset answers whether behavior that already worked remains intact.

And you need both to be successful. A candidate that fixes every known failure while breaking working behavior is not an improvement, and a failure-only dataset cannot tell you that.

id: rivn_missing_recommendation_001

input:

  tickers: [RIVN]

  focus: cash runway and investment conclusion

research_context: <captured research output>

expected_behavior:

  - report remains faithful to research

  - report gives a specific recommendation or next step

prohibited_behavior:

  - generic conclusion without a decision

human_label:

  actionability: FAIL

failure_category: lacks_actionable_guidance

source_trace_id: trace_abc123

risk_level: medium

Preserve only the state required to reproduce and judge the behavior. Remove secrets and sensitive user data before promoting production traces into a durable dataset — a dataset outlives the trace-retention window, so anything you copy into it becomes a longer-lived copy of that data. See masking and redaction for handling this before export, and build a dataset for creating and versioning the dataset itself.

Keep the experiment controlled

A useful experiment has three ingredients:

1. A fixed dataset
2. A task that runs one version of the agent
3. A fixed set of evaluators

# Illustrative structure. See the experiments docs for the current SDK surface.

baseline = run_experiment(

    dataset=agent_eval_dataset,

    task=financial_agent_v1,

    evaluators=[ticker_eval, faithfulness_eval, actionability_eval],

)

candidate = run_experiment(

    dataset=agent_eval_dataset,

    task=financial_agent_v2,

    evaluators=[ticker_eval, faithfulness_eval, actionability_eval],

)

Keep the dataset and evaluators unchanged while comparing the baseline and candidate. Changing the judge and the agent in the same step produces a number you cannot attribute to either. Then inspect examples that changed:

Aggregate scores provide orientation. Example-level diffs will tell you whether the change is actually better.

For the product mechanics, see set up an experiment and run evals on experiments. For running this comparison automatically on every change, see CI/CD for automated experiments and the conceptual treatment in pre-production LLM evaluation.

12. Move the same evals into production

Development evals become more valuable when the same criteria continue running after deployment. For reusable LLM-as-a-Judge and code evaluators, keep the evaluator definition stable while configuring the appropriate task, mappings, trigger, and sampling for the production data source. Agent-as-a-Judge follows a different runtime path: it is attached to an online eval task and its harness reads project traces at run time.

Use a simple production pattern:

1. Sample live traffic. Code checks avoid model-inference cost and are usually cheap enough to run broadly, but choose coverage based on data volume and computation rather than assuming 100 percent is always free. Sample costlier LLM- and agent-based judges according to risk, volume, and budget. Continuous evaluation tasks in Arize AX take an explicit sampling rate and a query filter, so you can score every span from a high-risk workflow while sampling a fraction of a high-volume one.
2. Alert on sustained changes. One stochastic failure may be noise. A rising failure rate within a task, tool, customer, or agent version is a signal.
3. Promote confirmed failures. Review the trace, confirm the expected behavior, remove sensitive data, and add the example to the regression suite.

Segment the results by workflow, tool, risk level, language, agent version, and failure category. A single average can hide a serious regression in a small but important slice.

For configuration, see run online evals on traces and production monitoring. For the conceptual difference between offline and online evaluation and what changes when the evaluator runs on live traffic, see production LLM evaluation and online versus offline evaluators.

13. Automate pattern discovery after you understand the loop

Manual trace review remains the foundation because it teaches you what evidence matters and how failures should be classified. Automating step 7 before you have done it once tends to produce categories nobody trusts.

Once that loop is working, automation can speed it up. Signal in Arize AX, for instance, is a built-in managed agent that scans a tracing project’s traces on a schedule, groups recurring failure patterns into ranked issues, and writes an investigation for each one with an overview, linked trace evidence, and a suggested fix. Issue detection is available on all Arize AX plans with monthly limits on Free and Pro; repo-backed fix pull requests are an Enterprise capability.

Treat any proposed change as a hypothesis. Review the evidence, run a controlled experiment, and decide whether the result meets the product requirement. Automation shortens the path from a production trace to an investigated issue. The requirement, release decision, and acceptance threshold still belong to the team.

14. Start with one trace and one eval

You do not need a large evaluation platform before the first useful step.

A practical first session looks like this:

1. Read traces for 15 minutes.
2. Write down one observable definition of success.
3. Choose one frequent or severe failure.
4. Add one code eval if the condition is objective.
5. Add one narrow LLM-as-a-Judge if the condition depends on meaning and the required evidence can be mapped cleanly.
6. Use Agent-as-a-Judge when the judgment requires runtime exploration of multi-step trace context.
7. Read every failed example.
8. Save confirmed failures and representative passes.
9. Rerun the set after the next change.

That workflow replaces “this looks better” with evidence tied to examples.

Agent-eval build checklist

Requirements

• The task outcome is observable.
• Required, allowed, and prohibited actions are defined.
• The failure-handling behavior is defined.
• North-star metrics and release guardrails are separated.

Tracing

• User input and final output are captured.
• Model, retrieval, and tool spans are connected.
• Tool arguments, results, and errors are available.
• Relevant state changes can be inspected.
• Session IDs connect multi-turn interactions.
• Prompt, model, and agent versions are recorded.
• Short-lived processes flush spans before exit.
• Sensitive fields are masked or redacted before export.

Error analysis

• Representative traces were read end to end.
• Observations were recorded before categories were defined.
• Failure categories describe causes rather than vague symptoms.
• Categories were ranked by frequency and severity.

Evaluators

• Objective conditions use deterministic checks.
• Standard LLM judges measure one stable semantic dimension each; Agent-as-a-Judge is reserved for multi-field or trajectory judgments that benefit from runtime trace exploration.
• Each judge has explicit evidence and labels.
• Agent-as-a-Judge instructions define what to score; fixed labels are used when the workflow needs stable aggregation, while open-ended labels are used deliberately.
• Judges use low-variance settings where supported, and repeatability is checked empirically.
• Missing evidence can produce an abstention or review route.
• Inapplicable examples are excluded from pass-rate denominators.
• Failed results include an inspectable explanation.
• Evaluator scope matches the evidence the criterion requires.

Judge validation

• Humans labeled examples with the same rubric and a fixed label schema.
• The calibration set reflects the base rate of the slice being scored.
• A held-out set was reserved.
• Precision and recall were measured per important label, with raw counts.
• Disagreements were reviewed by failure type.
• Pairwise judges were tested in both response orders.
• Failed examples generally seem fair to reviewers.

Datasets and experiments

• Confirmed failures were saved.
• Representative passing examples were also saved.
• Sensitive data was removed.
• Baseline and candidate use the same dataset and evaluators.
• Example-level changes are reviewed alongside aggregate metrics.

Production

• Evals run on an appropriate sample of live traffic.
• Alerts use sustained changes and meaningful slices.
• Confirmed production failures become regression cases.
• Judge behavior is periodically recalibrated.

Frequently asked questions about building agent evals

Should I write evals before I have production traffic?

Yes. Start with product requirements, manually tested scenarios, known edge cases, and synthetic inputs. Replace or supplement synthetic cases with real production traces as soon as they become available. Production traces tend to expose ambiguous, incomplete, and adversarial behavior that test authors did not anticipate.

How many traces should I read before writing the first eval?

Read every trace when the set is small. A dozen varied traces can be enough to find the first useful failure mode, although it is not enough to estimate production reliability. Keep expanding the dataset as new tasks and failures appear.

Can I build evals without traces?

You can evaluate a final response without a full trace, but you will be unable to distinguish many causes. A wrong answer may come from retrieval, tool selection, tool arguments, reasoning, state handling, or final synthesis. Traces make those failures diagnosable and let the evaluator inspect the evidence available at the moment of each decision. See AI agent tracing and evaluation for what to capture.

What is the difference between a code eval and an LLM-as-a-judge?

A code eval uses deterministic logic and should return the same verdict for the same inputs when it has no changing external dependencies, making it the right choice for schema validity, prohibited actions, thresholds, and recorded state checks. An LLM judge prompts a model to apply a rubric, which is useful when the criterion depends on meaning — faithfulness, actionability, or whether an escalation was appropriate. The practical test is whether two reviewers with the same rubric and the same evidence could disagree. If they could not, use code.

When should I use Agent-as-a-Judge instead of LLM-as-a-Judge?

Use a standard LLM-as-a-Judge when a fixed prompt plus mapped inputs contains the evidence needed for a stable, high-throughput semantic check. Use Agent-as-a-Judge when the evaluator needs to explore trace data at run time, reason across several spans or fields, or inspect a multi-step trajectory before scoring. Use code for deterministic rules. Agent-as-a-Judge writes standard eval results back to Arize, but its harness-based execution is different from a single LLM judge call.

Should an agent eval grade the outcome or the trajectory?

Make the outcome the primary criterion when several valid paths can solve the task. Evaluate the trajectory when intermediate behavior affects safety, permissions, policy, cost, latency, or external state. A correct answer does not excuse an unauthorized action, and an unexpected path should not fail merely because it differs from one reference sequence. For trajectory-specific strategies — exact match, partial ordering, required and forbidden actions, and semantic trajectory judges — see AI agent tracing and evaluation.

When should I use a span, trace, or session eval?

Use a span when the necessary evidence is local to one operation. Use a trace when the verdict depends on one complete request. Use a session when the verdict depends on several turns or requests. Select the smallest scope that contains the evidence you need. Trace and session evals may assemble more context and therefore cost more; use filters or multi-span queries to control which spans contribute to the judgment.

Should LLM judges use binary labels or numeric scores?

Use binary or categorical labels when the decision is discrete because the boundaries are easier to define and validate. Add insufficient_evidence or needs_review when the judge may lack context. Use numeric scores only when the underlying dimension has anchored levels and a labeled validation set shows that the scale behaves consistently.

How do I know whether an LLM judge is trustworthy?

Compare it with human labels created from the same rubric. Measure per-label precision and recall, inspect disagreements, test important slices, and reserve a held-out set. There is no universal minimum number of labels; use enough examples to estimate the error rates that matter for the action and to cover important slices. Continue auditing after deployment, because new failure modes, model changes, scoring-instruction changes, or data-mapping changes can all change its behavior.

Can I run the same eval in development and production?

For standard LLM-as-a-Judge and reusable code evaluators, yes: keep the evaluator definition stable and reuse it across datasets, experiments, and online tasks with the mappings and task configuration appropriate to each data source. Agent-as-a-Judge currently runs through online eval tasks over project traces, so do not assume the same harness-based evaluator is an offline experiment evaluator today.

What should become a regression test?

Promote a failure after the team confirms that the behavior is wrong, agrees on the expected result, and can preserve enough evidence to reproduce the case. Keep representative passing cases as well so improvements to one failure mode do not silently break working behavior.