Agent evaluation
Agent evaluation tests whether an AI agent completes its assigned task correctly and follows an acceptable path to get there. Effective evaluation goes beyond the final response to inspect tool calls, retrieval, state changes, constraints, and recovery behavior. This guide explains how to build representative datasets, trace complete runs, choose the right evaluators, compare changes, and turn production failures into regression tests.
This post is authored by Aryan Kargwal, PhD at PolyMTL.
What is agent evaluation?
Agent evaluation is the process of testing whether an AI agent can complete its assigned task correctly and behave as expected while doing so. It measures both the result and how the agent reached it.
An AI agent can make decisions, use tools, retrieve information, and take multiple actions before producing a result. Each step creates behavior that teams may need to test and inspect.
Agent evaluation gives teams a structured way to find failures, compare changes, and determine whether an agent is ready to handle real tasks reliably.
Why agent evaluation matters
Agent performance depends on more than the LLM or the Agentic framework it works on. Agent reliability depends on the full system around the model. Tool calls, retrieval, memory, permissions, and orchestration can each change whether the same model succeeds or fails.
Teams therefore need agent evaluation for several practical reasons:
- Agent behavior is stochastic. The same request can produce different plans, tool sequences, and outputs across runs. Teams need repeated evaluation to measure whether good behavior is consistent rather than occasional.
- Tool execution can fail independently of the model. An agent may select the wrong tool, generate bad arguments, or misread a response. Tool-calling evaluation tests whether those actions were actually correct.
- Production telemetry is often fragmented. Model calls, tool runtimes, APIs, and client infrastructure may live in different systems. Reconstructing a run requires connecting those signals before teams can judge what happened.
- Operational success does not guarantee semantic correctness. A request can complete without errors while the agent retrieves irrelevant evidence, misunderstands intent, violates policy, or reports an action that never occurred.
- Changes anywhere in the agent stack can introduce regressions. A new prompt, model, tool schema, memory strategy, or orchestration rule can alter behavior, even when application code around it stays unchanged.
Agent evaluation adds behavioral and semantic checks to the execution traces produced by the system. Teams can see where behavior diverged from expectations and verify whether a change actually fixed it.
What should you evaluate in an AI agent?
Agent evaluation should cover the outcome, the path taken to reach it, the actions performed along the way, and the context or constraints that shaped those decisions. Each surface can fail independently.
Did the agent complete the task?
Task completion should be verified against the system where the work actually happened. A support agent might confirm a refund that was never issued, while a coding agent might report a fix before the relevant tests pass.
The expected result depends on the task. Success may be a correct response, a verified state change, a completed workflow, or an artifact that satisfies defined requirements.
Did the agent take an acceptable path?
Two successful runs can follow very different trajectories. One may complete the task directly, while another loops through retrieval, repeats model calls, or invokes tools that were never necessary.
Those differences affect more than efficiency. Intermediate actions can introduce risk, change external state, or make a workflow fragile even when the final outcome is correct.
Some tasks require exploration, retries, or recovery. Evaluation needs to distinguish necessary steps from dead ends, repeated mistakes, and trajectories that succeed only because later actions compensate for earlier failures.
Agent-native evaluation goes deeper into trajectories, individual decisions, outcomes, and repeated-run behavior for agents whose performance cannot be judged from the final response alone.
Were the agent’s actions correct?
Tool calls can read sensitive data, modify external systems, trigger workflows, and create costs outside the model. Action-level evaluation therefore needs to inspect what the agent actually attempted.
Common failures include:
- selecting the wrong tool
- supplying incorrect or incomplete arguments
- inventing information required by the tool
- misreading the returned result
- acting outside the permitted scope
- continuing after a failed action
Wrong tool selection points toward routing or decision logic. Malformed arguments point toward parameter generation. Misread responses point toward how the agent interprets tool output.
Tool-calling evaluation separates tool selection from parameter generation, helping teams identify which part of an action failed instead of scoring the entire run as one unit.
Did the agent use the right context, state, and constraints?
Agents make decisions from retrieved documents, conversation history, memory, previous tool outputs, and application state. Incorrect or stale information can send later actions in the wrong direction even when the model reasons correctly.
Evaluation should verify that relevant information was available when decisions were made and that changes caused by earlier actions were reflected in later steps.
Constraints need the same scrutiny. Permissions, policies, approval requirements, and user-specific limits determine which actions are valid, even when the agent has the technical ability to perform them.
How to evaluate an AI agent
Agent evaluation works as a repeatable development loop: define expected behavior, collect representative tasks, capture complete runs, evaluate the behaviors that matter, compare changes, and turn observed failures into regression cases.
For a hands-on implementation, the Agent Trajectory Evaluations documentation guide walks through evaluating complete agent traces in Arize AX, from ordered tool calls to trajectory judging and logged evaluation results.
See the agent evaluation loop in practice 👀
Watch how the Arize team turns a production trace into an evaluation case, runs experiments against a proposed fix, and adds the failure to a growing regression suite.
Step 1: Define successful agent behavior
Define success before choosing evaluators. Record the result the agent should produce, the actions it may take, the constraints it must follow, and how it should respond when the task cannot be completed.
| Requirement | What to define | Evidence to inspect |
|---|---|---|
| Task outcome | The observable result that confirms completion | Final response, system state, or generated artifact |
| Agent actions | Required, allowed, and prohibited actions | Tool calls, arguments, and execution sequence |
| Constraints | Policies, permissions, limits, and approval requirements | Authorization state and policy checks |
| Failure handling | When the agent should retry, stop, or escalate | Errors, retries, and escalation events |
| Quality criteria | Correctness, completeness, and task-specific requirements | Evaluator results or human-reviewed labels |
Each requirement should point to evidence that can be captured during a run. Task holders and collaborating teams should agree on these conditions before prompts or models are compared.
Step 2: Build an evaluation dataset
Use a fixed set of tasks to compare agent changes over time. Start with common production requests, high-risk edge cases, and failures collected from traces, evaluator results, or human review.
Each row should preserve what is required to rerun the task: the input, relevant context, available tools, expected output or system state, and metadata such as task type or failure category.
A practical starting workflow in AX is:
- Filter traces for failed evaluations or problematic behavior.
- Select representative examples instead of exporting every failure.
- Add the expected outcome or human-reviewed label.
- Save the examples as a regression dataset.
- Add new production failures as the agent changes.
Let Alyx build the dataset for you. Press Cmd+L (macOS) or Ctrl+L (Windows/Linux) to open Alyx and try: “Create a dataset from the spans with errors” or “Create a synthetic dataset with 100 examples for regression testing”
AX also supports datasets created from CSV files, code, or synthetic examples. The Build a dataset guide walks through each route and shows how production spans can become reusable evaluation cases.
Step 3: Trace the agent’s behavior
Capture the complete path from the initial request to the final result. Read the trace for:
- Semantics: whether each decision and response fits the task
- Retrieval: what information was fetched, repeated, missed, or ignored
- Tool use: which tools were called, with what arguments, and how often
- Latency: where the run slowed down or waited unnecessarily
- Errors: failed calls, retries, timeouts, and failures carried into later steps
- Readability: whether engineers can reconstruct the run without piecing together disconnected logs
Step 4: Choose the right evaluators
An evaluator turns one requirement into a repeatable verdict. It should inspect the evidence that defines the failure and use the least subjective method capable of deciding it.
Use code when success can be verified through system state, tool arguments, execution order, latency limits, or policy conditions. Use an LLM judge when correctness depends on meaning, evidence quality, or decision context.
Human review is needed when evidence is incomplete, evaluators disagree, or an incorrect verdict could permit a high-consequence action.
For a deeper look at how evaluators are used across agent outcomes, trajectories, individual decisions, and repeated runs, read AI Agent Evaluation: An Agent-Native Framework.
Common evaluator patterns for AI agents
Different failures require different evaluators. Choose the pattern that matches the behavior you need to verify and the evidence available in the trace.
| Evaluator pattern | What it judges | Evidence to inspect | Typical evaluation method |
|---|---|---|---|
| Task-completion evaluator | Whether the agent completed the assigned job and produced the required result | Final response, generated artifact, test results, or state in the system where the action occurred | Code-based verification when the result is observable; LLM judge for semantic outcomes |
| Tool-selection evaluator | Whether the agent chose the correct tool for the task | Available tools, selected tool, user request, and decision context | Code-based matching for deterministic routing; LLM judge when several tools may be reasonable |
| Tool-argument evaluator | Whether the agent supplied valid, complete, and supported parameters | Tool schema, generated arguments, retrieved context, and application state | Schema validation, code checks, or exact field-level comparisons |
| Tool-result interpretation evaluator | Whether the agent correctly understood and acted on the tool’s response | Tool output, subsequent reasoning, later tool calls, and final response | LLM judge supported by deterministic checks for status codes or returned values |
| Trajectory evaluator | Whether the agent followed an acceptable path rather than looping, repeating mistakes, or taking risky unnecessary actions | Ordered model calls, retrieval steps, tool calls, retries, and state changes | Code checks for repeated or prohibited actions; LLM judge for overall path quality |
| Context-use evaluator | Whether the agent used relevant, current, and sufficient information | Retrieved documents, memory, conversation history, tool outputs, and application state | Retrieval metrics, citation checks, or an LLM judge for relevance and sufficiency |
| Constraint-compliance evaluator | Whether the agent followed permissions, policies, approval requirements, and task-specific limits | Authorization state, policy checks, tool calls, and attempted actions | Deterministic policy checks, rules, and code-based evaluators |
| Recovery and escalation evaluator | Whether the agent responded correctly when an action failed or the task could not be completed | Errors, retries, stop conditions, fallback behavior, and escalation events | Code checks for retry limits and required escalations; LLM judge for recovery quality |
| Response-quality evaluator | Whether the final answer is correct, complete, grounded, and appropriate for the request | Final response, source evidence, expected criteria, and human-reviewed labels | LLM judge calibrated against human-reviewed examples |
| Efficiency evaluator | Whether the agent completed the task within acceptable cost, latency, and execution limits | Token usage, latency, tool-call count, retries, and successful task completion | Deterministic thresholds and calculated metrics |
Step 5: Run evaluations and investigate failures
Treat every prompt, model, tool, or orchestration change as a separate experiment. Keep the dataset, evaluators, and run configuration fixed so the results remain comparable.
| Stage | What to do |
|---|---|
| Baseline | Run the current agent version and save the results |
| Candidate | Run the proposed change against the same dataset |
| Compare | Find examples whose evaluator labels, scores, latency, or cost changed |
| Investigate | Open the failed run and locate the first point where its trace diverged |
| Record | Assign a failure category and note the change most likely to address it |
Start with examples that changed from passing to failing rather than reading every trace. Review the output, evaluator explanation, tool sequence, retrieved context, and operational metrics for those rows.
In AX, Compare Experiments places outputs, evaluator results, and metadata side by side, and Diff Mode highlights how each run changed against a baseline you select. Alyx can also summarize the comparison with a request such as:
Compare my last two experiments and show where task completion fell, tool behavior changed, or latency increased.
The Run experiment guide covers the complete UI and code workflow. The Compare experiments guide shows how to inspect changes across individual examples.
Step 6: Add failures back into your evaluation suite
Do not copy every failed trace into the dataset. Confirm the failure, remove duplicates and sensitive data, then preserve the smallest amount of state required to reproduce and judge it.
Production trace → confirmed failure → regression case → experiment → deployment
Store each case in a consistent format:
input:
relevant_context:
available_tools:
expected_behavior:
observed_failure:
failure_category:
source_trace_id:
Add the case when the expected behavior is agreed and its supporting evidence can be inspected. Run the proposed fix against it before merging, then retain it in the regression suite for future changes.
After deployment, run the same evaluators against production traces. In AX, auto-add rules can capture spans by evaluation label, latency, token usage, or tool activity.
Use automatic collection for clearly defined conditions. Send ambiguous or high-risk failures through human review before promoting them into the permanent dataset.
Common mistakes in agent evaluation
Agent evaluations can return consistent scores and still give teams the wrong picture. The test cases may miss important behavior, the evaluator may judge the wrong evidence, or an average may conceal a serious failure.
Evaluating the output without checking the run
A correct-looking response does not prove that the agent completed the task correctly. It may have used the wrong source, called an unnecessary tool, ignored a failed action, or reported a state change that never occurred.
Check the final result against the system where the work happened. Then inspect the trace when the result and the agent’s reported outcome do not match.
Testing only clean examples
Hand-written examples usually describe tasks clearly and provide all the information the agent needs. Production requests are less controlled. Users omit details, mix multiple requests, provide conflicting context, and encounter tools or services that fail.
Keep common tasks in the dataset, but add ambiguous requests, high-risk workflows, failed tool calls, and confirmed production failures. Each case should represent behavior the team wants to preserve or prevent.
Trusting evaluators before calibrating them
An evaluator can apply an unclear rubric, reward a fluent answer, or reject valid behavior it did not expect. These errors become harder to notice once the evaluator is applied across thousands of runs.
Test the evaluator against examples reviewed by people who understand the workflow. Check where its verdict disagrees with those labels, revise the rubric, and repeat the comparison before using its score to approve releases.
Calibration should continue after deployment. New tasks and failure patterns may expose cases the original rubric did not handle.
Relying on aggregate scores
An overall pass rate can improve while one tool, workflow, or high-risk task becomes less reliable. The average hides which examples changed and how important those changes are.
Break results down by task type, tool, risk level, and failure category. When a score changes, inspect the affected examples and traces before deciding whether the new agent version is better.
Agent evaluation metrics
Agent metrics should support a decision, not fill a dashboard. Before running an experiment, define which result must improve, which failures are unacceptable, and which operational limits the new version must satisfy.
A release scorecard may look like this:
| Role in the decision | Metrics | What they tell you |
|---|---|---|
| Primary outcome | Verified task completion, resolution rate, test pass rate | Whether the agent completed the work it was assigned |
| Release guardrails | Critical-failure rate, policy violations, unauthorized actions | Whether the change introduced behavior that should block deployment |
| Failure diagnostics | Tool accuracy, grounding, escalation quality, retry rate | Where unsuccessful runs broke down |
| Operational constraints | Cost per successful task, P95 latency, tokens per successful task | Whether the improved behavior is affordable and deployable |
A candidate should not pass because its average score increased. It should improve the primary outcome without crossing a release guardrail or creating an unacceptable cost or latency regression.
Review these metrics by task type, tool, risk level, and failure category. Aggregate results can hide a serious regression inside a small but consequential group of tasks.
For detailed metrics by agent type, read Agent Evaluation Metrics: How to Measure Whether an Agent Works.
How Arize AX connects the agent evaluation loop
Arize treats agent evaluation as a debugging discipline. A failed evaluation should show what behavior failed, where it failed, and whether a proposed change fixes it.
Each verdict should remain connected to the run it assessed. Engineers need the evaluator explanation beside the model calls, retrieved context, tool results, and application state that produced the behavior.
This evidence is central to Arize’s evaluation harness. The evaluator produces the verdict, while the trace provides the information needed to understand and challenge it.
Failures also need to be reproducible. Once the expected behavior is confirmed, the run can become a test case for comparing changes outside production.
Arize’s approach extends this into full-agent experimentation. Teams can compare changes to the prompt, model, retrieval, tools, and orchestration instead of testing the model response alone.
A score that cannot lead back to evidence is difficult to debug. A production failure that cannot become a repeatable test is likely to return.
Agent evaluation in practice
Industrial agents rarely operate in a clean environment. At Axium Industries, they work across ERP records, warehouse data, equipment telemetry, and operational workflows.
A wrong decision can affect real inventory or machinery. Domain experts therefore help define the golden datasets, expected outcomes, and edge cases used to evaluate each workflow.
The team checks more than the final response. It evaluates task completion, tool use, consistency, and whether the agent produced the intended business result.
Axium initially built the supporting evaluation and observability infrastructure itself. As deployments grew, maintaining that stack began competing with the customer problems its engineers needed to solve.
Moving evaluation, tracing, and production monitoring to Arize Phoenix and the Arize platform reduced that burden. Engineers could spend more time improving the agents instead of maintaining the systems used to test them.
Arize uses a similar approach in its own AI-native support workflows.
Internal agents gather customer context, inspect traces and logs, reproduce issues, and prepare technical escalations. Tracing those runs exposed loops, incorrect tool calls, and steps that still required too much human intervention.
Median support resolution time fell from 22 hours in February 2026 to roughly 2.5 hours by June.
Related agent evaluation resources
- Agent Evaluation Concepts: Review evaluator patterns for agent planning, tool calling, and complete runs.
- Run Experiments With Code Evaluators: Define a task, apply code-based checks, and inspect evaluation results programmatically.
- Run Offline Evaluations on Experiments: Apply evaluators to saved experiments and compare their results with previous runs.
- View Agent Trajectories: Configure and inspect the execution path across agents, model calls, and tools.
- Evaluation Cookbooks: Explore runnable examples for trace-level and session-level agent evaluation, RAG retrieval quality, and more.
Evaluate your AI agents with Arize AX, or start with open-source observability and evals in Phoenix. Arize AX supports offline experiments, online evaluations over production traces, and agent trajectory judging.
Frequently asked questions about agent evaluation
How Is agent evaluation different from LLM evaluation?
LLM evaluation often scores a single model response for correctness, relevance, groundedness, safety, or style.
Agent evaluation also examines the work performed before that response. It may inspect planning, retrieval, tool calls, state changes, recovery behavior, and whether the assigned task was actually completed.
What should an agent evaluation dataset include?
Include common production tasks, high-risk workflows, ambiguous requests, tool failures, and confirmed production failures. Each case should preserve the input, relevant context, available tools, expected outcome, constraints, and enough state to reproduce the run.
Can an LLM judge evaluate an AI agent?
An LLM judge can assess behavior that depends on meaning or context. Examples include whether the agent followed instructions, used sufficient evidence, or chose a reasonable path.
It should not replace checks that can be verified directly. System state, tool arguments, execution order, required fields, and policy conditions are usually better evaluated with code.
Test judge verdicts against human-reviewed examples before using them for release decisions. Review both cases it incorrectly passes and cases it incorrectly rejects.
How do you evaluate an agent when several paths are valid?
Evaluate the requirements shared by every acceptable run rather than enforcing one exact sequence. Tool order, task decomposition, and recovery strategy may vary as long as the agent reaches the required outcome, respects constraints, and avoids prohibited actions.
Should agent evaluations run before or after deployment?
Both serve different purposes.
Offline evaluations compare prompts, models, tools, and agent logic against controlled datasets before release. They make candidate versions easier to compare under the same conditions.
Online evaluations inspect production traces. Real users, changing data, external services, and unfamiliar requests often reveal failures that were absent from the original dataset.
What metrics matter most for agent evaluation?
Use a primary metric that verifies whether the agent completed its assigned job. Depending on the workflow, this could be resolution rate, successful state change, test pass rate, or completed workflow rate.
Add release guardrails for critical failures and diagnostic metrics for tool accuracy, grounding, retries, latency, and cost. These metrics explain the outcome but do not replace it.
How do you know an AI agent is ready for production?
The agent should meet its required outcome across representative, difficult, and high-risk tasks. It should also remain within defined safety, policy, cost, and latency limits.
Review repeated runs rather than relying on one successful attempt. Stochastic behavior can make a weak agent look reliable when the dataset or run count is too small.
Production readiness also depends on what happens after launch. Teams need complete traces, online evaluations, and a process for turning confirmed failures into regression cases.
Congratulations, you’ve reached the end of Arize’s Agent Handbook! 🎉
You’ve learned what agents are, how they’re made, different agent architectures and organization structures, and how to evaluate your agents.
If you have any questions, or just want to share what you’re working on, come join our team in Slack.
Now, go forth and build!