A polished answer can be the last thing an agent gets right.
Behind it, the agent may have called the wrong tool, changed nothing, or bypassed a required approval. Final-answer scoring sees only a small part of that execution.
A useful evaluation must also determine whether the agent reached the correct state, followed an acceptable path, made defensible decisions, and behaved reliably across repeated runs.
Agent-native evaluation treats the run as a stateful sequence of decisions and actions. It follows the execution, checks the resulting state, and asks whether the agent behaved acceptably throughout the run.
Agent-native evaluation is not an attempt to specify every valid way an agent could complete a task. In practice, teams begin with representative scenarios that can be verified reliably, then expand coverage as new failures and behaviors appear in production.
The rest of this article develops that framework around the failures agents actually produce, from controlled experiments to live production traces.
TL;DR: What agent-native evals can and cannot do
Agent-native evals assess the outcome an agent produced, the path it took, the decisions it made, and whether that behavior remains reliable across repeated runs. They do not attempt to enumerate every valid way an agent could solve a task.
- Start with representative, verifiable scenarios, particularly frequent or high-consequence failures. Open-ended and multi-goal tasks may have many valid trajectories.
- Define required outcomes, invariants, and boundaries rather than one “correct” sequence. Evaluation criteria should be precise enough to verify without penalizing equally valid approaches.
- Keep evaluators as implementation-agnostic as possible. Checks tied directly to tool names, span layouts, or trace schemas require additional maintenance as the agent’s implementation changes.
What is agent-native evaluation?
Here, native means designed around the agent’s operating model. Agents observe a state, choose an action, receive a result, and use that result to decide what happens next.
General agent evaluation can score tool selection, parameters, plans, or responses. An agent-native evaluation preserves the relationships between those events rather than treating each as an independent output.
That context changes the verdict. A tool call may be correct after one observation, premature after another, or unsafe because an earlier action changed the agent’s permissions or available options.
Success is judged against the task’s behavioral specification across the full trajectory. Multiple paths can pass when they reach an acceptable state while respecting required actions, prohibited actions, and operating boundaries.
The evaluator should therefore encode invariants rather than a single ideal trajectory: required state changes, prohibited actions, policy boundaries, and evidence that must be present. Ordering, task decomposition, and recovery strategy can remain flexible when several approaches are equally valid.
Why final-answer evaluation misses most of an AI agent run
Final-answer scoring begins after most of the agent run has already happened. Before responding, the agent may reason through alternatives, inspect external state, call APIs, retry tools, revise its plan, and modify connected systems.
The customer usually sees none of this execution. A convincing response can follow an inefficient, unsafe, or partially failed run. A failed response can also hide substantial progress and a recoverable error near completion.
Consider Terminal-Bench, a staple benchmark for terminal-based agents. Agents complete realistic tasks inside command-line environments, while comprehensive tests verify the environment they leave behind rather than the explanation they provide.
That design captures something final-answer grading misses: task completion lives in the resulting state. Intermediate milestones can further reveal partial progress, stalled execution, and the point where an otherwise promising run broke down.
Recent agent-evaluation literature points toward several broader trends:
- More reasoning tokens can increase cost and latency without producing a better result.
- Tool retries separate successful recovery from blind repetition and unresolved failure.
- API responses may be incomplete, stale, misinterpreted, or disconnected from the resulting system state.
- Plausible individual actions can still form an inefficient, unsafe, or policy-violating trajectory.
- Repeated runs expose unstable behaviour that a single successful answer cannot establish.
The same pattern appears in enterprise deployments. Customers see the message; the trace reveals whether the agent completed the work safely, efficiently, and consistently.
The four scopes of AI agent evaluation
Once the response is separated from the execution behind it, agent evaluation breaks into four distinct scopes. Each answers a different question about whether the run should be considered successful.
Teams must examine the outcome reached, the path taken, individual decisions, and reliability across repeated runs.
Together, they turn a pass/fail result into an actionable diagnosis: whether the task was completed, where execution diverged, which decision caused the failure, and whether the behaviour is stable enough for production.
Evaluating whether the agent reached the right outcome
Outcome evaluation checks whether the agent left the task in an acceptable final state. That state may live in a database, ticketing system, codebase, generated document, or physical environment.
Consider a refund agent. Success requires the correct order to be refunded for the approved amount, the customer record to be updated, and the case to leave the unresolved queue.
The agent’s confirmation message cannot establish those conditions. The evaluator must inspect the affected systems after execution and compare their actual state against explicit completion criteria.
Structured outcomes can usually be checked with deterministic assertions. Teams can validate the refund amount, confirm that the record was updated, and verify that the support case was closed.
These checks can also represent partial completion. A recorded refund with an open case receives a different verdict from a run that produced no valid state change.
Less-structured outcomes, such as reports or customer communications, may require semantic evaluation alongside checks that the correct source data was retrieved and persisted.
Evaluating the agent’s trajectory
A correct outcome does not prove sound execution. An agent may reach the requested state after unnecessary retries, unsafe commands, skipped approvals, or access to unrelated resources.
Trajectory evaluation examines the sequence of observations, decisions, tool calls, and state changes that produced the result. It checks whether required steps occurred, actions respected policy, and the agent remained within its assigned authority.
Consider an agent asked to replace a part, refund an earlier order, and place a new order. A trajectory evaluator can determine:
- Whether the agent completed each of the user’s sub-goals
- Whether it called the appropriate tools in a valid sequence
- Whether it introduced redundant steps, retries, or loops
- Whether it recovered appropriately from tool failures
- How much latency, cost, or token usage the task required
These questions require evaluating the trajectory as a whole. An individual tool call may appear valid in isolation while contributing to an incomplete, inefficient, or unsafe run.
For tasks with a known workflow, teams can also use a reference-based trajectory evaluation. The evaluator compares the observed run against required steps or a set of acceptable paths. For a refund task, those steps might include verifying the order, checking the refund policy, obtaining approval when required, processing the refund, and confirming the resulting state.
The reference should generally define required actions and boundaries rather than mandate one exact sequence. Several trajectories may be valid unless a particular order is necessary for safety, policy compliance, or task completion.
Open-ended systems such as OpenClaw and Hermes Agent make trajectory evaluation especially important. They can inspect environments, invoke tools, modify files, and revise their plans as new information appears. Their usage also provides a live view into the work people are beginning to delegate to agents.
In enterprise deployments, teams already reconstruct parts of this trajectory in the form of traces from self-hosted systems or vendor APIs. Tool logs, permission prompts, command summaries, and progress updates expose what the agent attempted.
That visibility requires sufficient instrumentation. Low-level actions must be connected to the task, prior state, available evidence, expected effect, and resulting consequence. A command or API call should be judged according to why it was issued, what resource it targeted, what authority the agent held, and what happened afterward.
Guardrails built around that context can distinguish useful recovery from blind repetition, valid exploration from scope creep, and necessary execution from risky behavior. Trajectory evaluation establishes whether the agent completed the work through an acceptable and efficient process.
The most important adjustment to the reviewer’s proposed reference-based example is using “required steps or acceptable paths” rather than one predefined path. That preserves the earlier caveat that agents can complete complex tasks in several equally valid ways.
Evaluating individual agent decisions
Decision evaluation examines one choice within the run. It asks whether the action was supported by the evidence, policy, permissions, and alternatives available when the agent acted.
Return to the refund agent from the earlier example. It detected that the requested amount exceeded its automatic approval limit, yet continued processing and later told the customer that the refund succeeded.
At that point, the agent had several valid alternatives:
- Request approval
- Escalate case
- Request review
- Stop execution
The evaluator reconstructs the decision context: what the agent observed, which actions were available, what each action could change, and which constraints applied at that moment.
Clear violations can be checked deterministically. A rule can fail any refund issued above the agent’s authority or any success message sent before the refund status is confirmed.
Ambiguous choices often require an LLM-as-a-judge with a narrow rubric. The judge compares the selected action against the available evidence and returns a verdict with a reason.
An agent-as-a-judge can go further by inspecting the surrounding trace, retrieving policy details, checking external state, and using tools before deciding whether the action was justified.
This scope identifies the exact decision that introduced the failure. Teams can then change the prompt, approval policy, tool permissions, or decision logic responsible for that choice.
Measuring reliability across repeated runs
A successful run shows that the agent can complete the task once. Reliability evaluation measures whether acceptable behavior persists across repeated attempts and realistic variations.
The same refund request may produce different results. One run may request approval, another may exceed the limit, and another may confirm success before checking the refund status.
Teams should therefore repeat representative tasks and compare outcome consistency, policy-compliance rate, recovery behavior, critical failures, latency, and cost across runs.
Average success can still hide rare unauthorized actions or unstable execution costs. Reliability thresholds should therefore reflect both failure frequency and operational consequence.
Reliability can be measured through pass@1, pass@k, and pass^k. Pass@1 measures whether the agent succeeds on a single attempt. Pass@k measures whether at least one of k attempts succeeds, capturing whether the agent can complete the task given multiple opportunities. Pass^k measures whether all k attempts succeed, providing a stricter measure of consistency. Teams can also track prompt robustness, resource variance, policy-compliance rate, and critical-failure frequency.
Each metric answers a different operational question. Pass@k rewards recoverability, pass^k demands consistency, and severity-weighted failure metrics expose risks that an average success rate would conceal.
How to write an evaluation specification for an AI agent
The four evaluation scopes become operational only when they are written into an evaluation specification. The specification defines acceptable outcomes, permitted paths, decision boundaries, and reliability requirements before execution begins.
Using the refund example, the specification can be expressed as a compact executable card. Each field maps to a scope and identifies the evidence required to judge the run.
evaluate_refund(run):
outcome_ok =
run.refund.order_id == run.request.order_id
and run.refund.amount == run.approved_amount
and run.customer_record_updated
and run.case_status == "resolved"
path_ok =
run.order_checked
and run.policy_checked
and run.refund_status_confirmed
and not run.approval_skipped
decisions_ok =
refund_processed only_if approval_received
and success_reported only_if refund_status == "completed"
reliability_ok =
policy_compliance_rate >= required_threshold
and critical_failure_rate <= allowed_threshold
if outcome_ok and path_ok and decisions_ok and reliability_ok:
return PASS
if run.refund_status == "completed" and run.case_status != "resolved":
return PARTIAL
return FAIL
The evaluator cannot derive these conditions from the agent’s final message. It must inspect the refund record, customer record, case queue, approval state, and execution trace.
The specification can allow several valid paths while preserving strict boundaries. The agent may request approval or escalate the case, but it cannot exceed its authority or report an unconfirmed refund.
How to choose the right evaluator for each agent failure
An evaluator can only produce a trustworthy verdict when it can observe the evidence that defines the failure.
A checker that sees only the final response cannot determine whether the agent changed the right system state. A model-based judge is equally unsuitable when a policy condition can be verified exactly.
Matching the evaluator to the failure source keeps the verdict anchored to the part of execution that actually broke. The evaluator should use the most direct evidence and the least subjective method capable of deciding the case.
When the available evidence cannot support a reliable automated verdict, the system should defer rather than force a pass or fail.
Evaluators should also depend on stable task semantics wherever possible. Checks against resulting state, policy conditions, or normalized events are generally more durable than checks tied to a specific tool name, span layout, or vendor-specific trace field. Some implementation coupling is unavoidable, but it creates maintenance whenever tools, schemas, or instrumentation change.
State and action failures should be checked deterministically
Deterministic evaluators work when the expected state or permitted action can be expressed as an explicit condition. They compare recorded evidence against that condition and return a reproducible verdict.
These evaluators work when the expected state or permitted action can be expressed as an explicit condition. The check could be a Python assertion, a database query, a schema or policy validator, or an LTL property evaluated over the execution trace.
For the refund agent, one check can verify that the stored amount matches the approved amount, while another confirms that approval existed before execution. Each failed assertion identifies the exact state transition or action that violated the evaluation specification.
| Failure source | Evidence inspected | Deterministic check |
|---|---|---|
| Incorrect resulting state | Persisted refund and case records | The refund matches the approved amount and the case is resolved |
| Invalid tool action | Tool call and submitted arguments | The refund targets the requested order and uses the approved amount |
| Skipped prerequisite | Approval record and execution order | Approval exists before a refund above the automatic limit |
| Prohibited action | Permission state and execution trace | The agent never processes a refund outside its authority |
These checks can run after controlled experiments or against production traces. Because each verdict is tied to an explicit condition, failures remain easy to reproduce, diagnose, and convert into regression tests.
Deterministic checks become insufficient when correctness depends on meaning, evidence quality, or the context surrounding a decision.
When to use an LLM judge for agent evaluation
Semantic evaluation is needed when correctness depends on interpretation, such as whether an explanation is faithful, a decision is justified, or a response addresses the user’s request. Broad instructions such as “decide whether this run was good” leave too much of the standard to the judge. Each evaluator should isolate one property, define the evidence it may use, and specify what qualifies as a pass.
- Pointwise evaluation: Judges one output against a fixed rubric and returns an independent score or verdict. It works well when every run must satisfy the same production or regression standard.
- Pairwise evaluation: Compares two outputs for the same task and determines which better satisfies the rubric. It is useful for selecting between prompts, models, or agent configurations, although the preferred output may still fail the required standard.
- Reference-guided evaluation: Gives the judge an authoritative policy, answer, trace, or set of acceptable examples. The judge checks the output against that evidence, making unsupported claims and missing requirements easier to identify.
The judge should cite the evidence behind its verdict and identify the criterion that failed. Teams should calibrate the rubric against human-labelled cases, inspect recurring disagreements, and repeat that calibration whenever the judge model or evidence pipeline changes.
Human review belongs at the boundary of uncertainty and consequence
Human reviewers can apply the same pointwise, pairwise, and reference-guided protocols used by model-based judges. Their role is to establish reliable labels, resolve ambiguous verdicts, and detect errors that automated evaluators consistently miss.
In practice, human judgment enters the evaluation pipeline through three recurring review patterns:
- Calibration review: Domain experts label a representative set of runs before deployment. Teams compare automated verdicts against those labels to refine the rubric, judge prompt, examples, and decision thresholds.
- Disagreement adjudication: A reviewer resolves cases where evaluators disagree, abstain, or produce incompatible rationales. The adjudicated verdict becomes the authoritative label for that case.
- Ongoing audit: Teams periodically sample automated passes and failures to detect evaluator drift, systematic bias, and confident errors that disagreement-based routing may overlook.
Review should also be mandatory when an incorrect verdict could permit a high-consequence action. Each reviewed case should preserve the criterion, evidence, final verdict, and rationale so it can become a calibration example or regression test.
Running agent-native evals in Arize AX
The framework described here depends on retaining the evidence behind every verdict. Arize AX tracing captures each run as connected spans containing inputs, outputs, tool calls, state changes, latency, and cost. Evaluators can then operate at the span, trace, session, or experiment level.
Teams can implement their evaluation specification with code evaluators for explicit conditions, LLM-as-a-judge for narrow semantic criteria, and Agent as a Judge when the verdict requires broader trace context. Human review provides the labels needed to calibrate those evaluators and resolve disputed cases.
The same criteria can run online against production traces and offline against controlled experiments. A production failure can become a dataset example, a candidate evaluator, and a regression case used to test the next prompt, model, retrieval, or agent-logic change.
For researchers, this creates a repeatable environment for comparing agent behaviour across configurations. Enterprise teams can carry the same evaluation specification from development into production. Start with one agent and one known failure: instrument the run, encode the criterion, and test the next change in Arize AX. Teams that need an open-source, self-hosted foundation can follow the same workflow with Phoenix.
Frequently asked questions
How are agent-native evals different from traditional LLM evals?
Traditional LLM evals commonly grade a generated answer for properties such as correctness, relevance, faithfulness, or style. Agent-native evals must also inspect what happened before that answer was produced.
Agent-native evaluation checks the execution trace and resulting environment so a polished response cannot conceal an incomplete or unacceptable run.
How do you evaluate an AI agent when multiple trajectories are valid?
Evaluate stable requirements rather than requiring one exact sequence of actions. These requirements may include the resulting state, required prerequisites, prohibited actions, policy boundaries, and evidence available at important decisions. The evaluator should allow tool order, task decomposition, and recovery strategy to vary when several approaches can complete the task safely.
What should teams evaluate in an AI agent run?
A complete agent evaluation should cover four scopes:
- Outcome: Did the agent produce the required final state?
- Path: Did it follow an acceptable and policy-compliant process?
- Decision: Were individual actions justified by the available evidence and permissions?
- Reliability: Does acceptable behaviour persist across repeated runs and realistic variations?
These scopes identify whether the task failed, where execution diverged, and which system component may need to change.
When should agent evaluations use deterministic checks or LLM judges?
Deterministic checks are preferable when success can be expressed as an exact condition. Examples include verifying a database update, validating tool arguments, confirming that approval preceded an action, or checking an execution trace against a policy rule.
LLM judges are better suited to semantic questions, such as whether an explanation is faithful or a decision was reasonable given ambiguous evidence. Each judge should evaluate one narrowly defined property using a clear rubric and specified evidence. High-consequence or genuinely uncertain cases should be routed to human review.
Why are traces important for evaluating AI agents?
Traces connect an agent’s inputs, observations, tool calls, outputs, state changes, latency, and cost into a coherent execution record. Without that context, evaluators may see that an action occurred without knowing why it was selected or what consequence it produced.
Trace-based evaluation can identify skipped prerequisites, unnecessary retries, unauthorized actions, poor recovery, and the precise decision that introduced a failure. The same trace can later become a regression test for evaluating a new prompt, model, tool, or agent configuration.
How can teams measure whether an AI agent is reliable?
Teams should repeat representative tasks and measure outcome consistency, policy compliance, recovery behaviour, critical failures, latency, and cost across runs. A single successful execution shows capability, while repeated evaluation shows whether that capability is stable enough for production.
Metrics such as pass@1, pass@k, and pass^k answer different questions. Pass@k measures whether at least one attempt succeeds, while pass^k measures whether every attempt succeeds. Severity-weighted failure rates are also useful because average success can conceal rare but consequential actions.