Agent-run evaluation is the practice of scoring a single complete execution of an agent: everything that happened between the triggering input and the terminal state, including tool calls, retries, intermediate reasoning, and the state the agent carried forward. The unit under test is the run, not the last message it produced.
That distinction carries most of the weight. A conventional LLM eval takes an input and an output and asks whether the output is good. An agent arrives at its output through a path that may have involved nine tool calls, two failed API responses, and a retry loop. Two runs can return an identical answer and be worth very different amounts: one resolved the account in a single lookup, the other searched four times, guessed at a customer ID, and happened to be right.
The run is also the unit your on-call engineer works in. Nobody reports a failure as “the correctness score dropped.” They report it as “the agent did the wrong thing for this customer at 3pm.”
Key takeaways
- A run is one full execution of an agent, from triggering input to terminal state. It is the unit production failures get reported in and the unit most agent evals should score.
- Run-level scores answer whether the task got done. Step-level scores answer where it went wrong. Both come from the same trace.
- A run that produced the right answer through a bad path should often fail, because the path is what breaks on the next input.
- Terminal state is its own signal.
completed,max_steps_exceeded,tool_error, andhanded_offare outcomes a single correctness score flattens into one number. - Judging every run with an LLM does not survive production volume, so deterministic checks run everywhere and judge scoring runs on a sample plus anything that already looks suspect.
What counts as a run
A run is one execution of the agent loop. The trace is the telemetry record of that run, a tree of spans covering model calls, tool invocations, and retrieval steps. A session is a series of runs with the same user. Settling these three levels before you define metrics saves rework later, and it is the first thing to fix when wiring tracing and evaluation together for an agent.
Where the boundary falls depends on the product: one turn for a chat agent, the whole task for a coding agent or a background job. What matters is that it stays stable. Redefine a run mid-quarter and every rate and trend you have computed becomes incomparable.
What you score at the run level versus the step level
At the run level, the questions are about outcome:
- Task completion. Did the agent accomplish what the user asked, as opposed to producing a plausible response about it.
- Final output quality. Correctness, groundedness in what the tools returned, format, policy adherence.
- Terminal state. Whether the run finished, gave up, hit a ceiling, or died on an unhandled tool error.
- Path cost. Step count, tool calls, tokens, wall clock. Not quality scores, but they describe the run as an operational object.
- Constraint adherence. Whether the agent stayed inside its permissions and skipped write actions during a read-only task.
At the step level, the questions are about mechanism: was the right tool selected, were its arguments well formed, did the retrieved context support the claim made after it, and did the agent recover from a tool error or plow ahead as if nothing happened.
Neither reduces to the other. A run can consist entirely of defensible steps and still fail, because no step was wrong but the plan was. Score only the run and you know that something failed but not where.
When the right answer is not a pass
The most useful thing run-level evaluation does is let you fail a run that got the answer right:
- The agent guessed a required parameter instead of looking it up, and the guess happened to match.
- The agent brute-forced thirty searches. It passes in a test with no time limit and times out in production.
- The agent answered from a stale cached document rather than the record it was supposed to read.
- The agent called a write endpoint during a read-only task. Nothing broke this time.
None of these are visible if you score the final answer, and all of them are ordinary in production. They belong to the same class of failures that conventional software tests miss, for the same reason: nothing errored and the output looked fine.
The counterweight is that not every inefficient path is a failure. Agents explore, and a run that took six steps instead of four is often just a run. Report path quality as a separate dimension rather than folding it into task success, so you can tell an agent that was wrong from an agent that was wasteful.
Sampling and cost
A judge that evaluates a run has to read the run, and a long trajectory is a long context. Run-level judging costs more per record than scoring an input and output pair, which is worth knowing before committing to full coverage and discovering where evaluation costs accumulate on the invoice.
The arrangement that holds up: deterministic checks on every run, since terminal state, step count, and forbidden tool calls cost nothing to compute; judge scoring on a stratified sample so low-volume entry points stay represented; and targeted scoring of any run that already looks wrong, meaning an error span, a retry, a hit step limit, or a user who immediately rephrased. Segment before you average, because a single aggregate mixes a two-step lookup with a forty-step research task and describes neither.
FAQ
What is an agent run?
One complete execution of an agent, from the input that started it to the point where it stopped, including every model call, tool call, and retry in between. In telemetry it appears as a trace, and in most chat products a run corresponds to a single user turn.
Does agent-run evaluation mean evaluation performed by an agent?
The phrase gets read both ways, so it is worth naming. This entry uses the reading that matches how teams talk about their systems: an agent run is a noun, and agent-run evaluation is evaluating one. The other reading, where an agent executes the evaluation workflow itself, is the pattern usually called agent-as-a-judge when it is scoring, or an autonomous evaluation system when it runs the whole loop.
Should a run pass if the final answer was correct?
Not automatically. Correct output through an unsafe or unrepeatable path is a run that will fail on the next input. Report task success and path validity separately, then decide which combinations you are willing to ship.
How many runs do I need to score before the numbers mean anything?
Enough that your less common paths appear more than once. The failure mode is a sample that is large in total but holds three examples of the workflow you are worried about. Sample by segment, and treat a rate computed from a handful of runs as a reason to read them individually rather than as a metric.
