Agent-native evaluation is evaluation designed around how agents actually run, rather than evaluation designed for single model calls and then pointed at an agent. It scores trajectories instead of final answers, understands that a run is made of steps, reads what the tools returned, and knows what the agent knew at each point in the run.
The contrast is concrete. A conventional LLM eval takes an input, an output, and sometimes a reference, and returns a score. Applied to an agent, it can tell you the final response looked reasonable. It cannot tell you the agent called the wrong tool, retried four times, skipped the verification step, or answered from context it retrieved for a different question. All of that happened inside the run, and the eval never saw inside the run.
It also changes what the results are for. Scores stop being a report a human reads on Friday and become data attached to spans, addressable by trace ID, queryable by a CI job or by another agent asking which cases regressed after a change.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Agent-native evaluation scores the trajectory: steps, tool calls, state, and terminal condition, not only the text at the end.
- Tool results are a form of ground truth available at runtime, which lets many agent evals run reference-free where a labeled dataset would otherwise be required.
- Evaluation has to respect state. Judging a step using information the agent only obtained later is a common mistake that produces unfairly harsh and unactionable scores.
- Results belong on the spans they describe, in a stable schema, so a bad score leads to the exact step that caused it rather than to a dashboard.
- The term is used loosely. The test that separates substance from positioning is whether the evaluation can express a judgment about an individual step.
What makes an evaluation agent-native
It is trajectory-level. The evaluated unit is a run: the ordered sequence of model calls, tool invocations, and observations between the triggering input and the terminal state. That sequence is the trace, which is why agent evaluation and tracing get built as one capability rather than two. The trace is not debugging output here, it is the input to the evaluator, and agent evaluation covers the instrumentation that makes trajectory scoring possible at all.
It is step-aware. It can attach a judgment to a single step: was this the right tool to call, were the arguments built correctly from what was available, did the agent recover after the tool returned a 500, was the retrieved chunk relevant to the sub-question being asked. Aggregate quality without per-step attribution tells you something is wrong and nothing about where.
It reads tool calls and their results. This property has the largest practical consequence. When an agent queries an order system and gets back a status, that response is ground truth for anything the agent says next about the order. An evaluator that can see the tool result checks the claim directly, with no labeled dataset and no human reference answer. Many of the most useful agent evals are reference-free for this reason.
It is state-aware. An agent accumulates state: retrieved documents, prior tool outputs, plan revisions, memory. Evaluating step four requires knowing what the agent had at step four. Score it against information that only arrived at step nine and you will mark correct behavior as a failure and send someone to fix a prompt that was fine. Agent failures attributed to the model routinely turn out to be problems with the context and the harness around it, which only becomes visible when the evaluation looks at what was in the window at the moment of the decision.
Its results are addressable. A score written back to the span it describes can be filtered, aggregated by tool or step type, and opened directly. A score in a separate report cannot. Stable schemas and programmatic access are what let results feed CI, monitors, and agent-driven workflows rather than only a human reviewer. The sibling terms “evaluation as infrastructure” and “evals as APIs” cover that platform contract in depth.
What agent-native evaluation replaces
The default alternative is a final-answer eval over a fixed input and output set. It stays useful: cheap, stable, and it catches genuine regressions in the text users read. It just cannot see most of what goes wrong in an agent. The failures it misses are the ordinary ones:
- The agent completed the task with a side effect nobody wanted.
- The agent looped between two tools and only stopped because it hit a step ceiling.
- The agent got the right answer by guessing a parameter.
- The agent silently degraded, answering from its own weights after retrieval returned nothing.
- The agent handled the request but took nineteen steps to do it, and the cost per resolution tripled.
That set overlaps heavily with the failures conventional software tests do not catch, for the same underlying reason: the system did not error and the output looked correct.
Where the term gets stretched
“Agent-native” is a positioning word about as often as a technical one. Three questions separate the two:
- Can the evaluation attach a score to a specific step, or only to the run as a whole?
- Can it read tool call arguments and tool responses, or only the model’s text?
- Do results land on spans in a queryable store, or in a separate report?
If the answers are no, it is a text eval with an agent-shaped label. That is not useless, and for some workloads it is the right cost trade-off, but it should not be called agent-native.
The honest limitation on the other side is that trajectory-level evaluation is more expensive and noisier. Judging a run means reading the run, which is a long context, and step-level scores multiply the judgments per record. Most teams land on a tiered arrangement: deterministic step checks everywhere, run-level judging on a sample, and step-level judging only on runs that already look wrong.
FAQ
How is agent-native evaluation different from standard LLM evaluation?
Standard LLM evaluation scores an output given an input. Agent-native evaluation scores a run: the sequence of steps, the tools called, the arguments passed, the results returned, and the state at each point. The first answers whether the text was good. The second answers whether the agent behaved correctly, which is a different question and usually the one that matters in production.
Can I reuse my existing LLM evals for my agent?
Partly. Output-level evals for groundedness, correctness, tone, and safety still apply to the agent’s final response and are worth keeping. What they will not give you is tool selection accuracy, argument validity, error recovery, path efficiency, or terminal state, and those are where agent-specific failures concentrate.
Does agent-native evaluation require labeled data?
Less than you would expect. Because tool responses are observable at runtime, many checks are reference-free: whether the answer matches what the tool returned, whether a required verification step ran, whether the arguments were derivable from available context. You still want a small labeled set to check that your judges agree with human judgment, but you do not need a labeled reference for every case.
What is the difference between agent-native evaluation and agent observability?
Observability produces the record of what the agent did and makes it searchable. Evaluation adds a judgment about whether what it did was acceptable. They share storage and a data model, which is the practical reason agent-native evaluation is defined partly by where its results live.
