How to evaluate whether your AI agents are actually improving

Last reviewed September 18, 2026.

Why AI agent evaluation scores can hide production failures

Most teams shipping agents can’t answer the question rigorously, and the reasons are structural rather than a failure of effort:

• Confidence comes from anecdotes. The demo went well, the founder’s test query worked, and a customer said something nice, none of which samples the traffic where problems live.
• The feedback signal is thin. Thumbs ratings capture a sliver of sessions, skew toward extremes, and say nothing about the majority of interactions, including the ones where a plausible answer hid a failed process.
• The denominator is dishonest. When failure discovery depends on customer complaints, the measured failure rate reflects who bothered to complain rather than what actually failed, so the metric improves whenever customers give up quietly.
• Changes ship without a before and after. A prompt tweak or model swap goes out, the team watches for smoke, and nobody can say a month later whether it helped, hurt, or did nothing.
• The ruler keeps changing. Teams that iterate on their evaluators without versioning them can’t compare this quarter’s scores to last quarter’s, since the judge changed along with the agent.

An agent can look great through every one of those lenses while its task completion rate drops for a customer segment nobody’s watching. The fix starts with defining what improvement means.

What does AI agent improvement actually mean?

For a chatbot, quality mostly means answer quality. For an agent, quality means the work got done, so improvement lives in a layered set of outcomes, and each layer answers a different question:

• Business outcomes. Did the agent move the number it exists to move, whether tickets resolved without escalation, tasks completed without human takeover, or revenue-relevant actions finished correctly? This is the layer a leader reads first.
• Session outcomes. Did the full session accomplish what the user came for, coherently, without losing context or completing the wrong version of the task?
• Trajectory quality. Did the agent take a reasonable path to get there, choosing the right tools with the right inputs, following the policies that matter, without loops and retries burning cost?
• Component health. Are retrieval, tool calls, latency, and spend behaving underneath it all?

The layers have to connect, because a business outcome that dips is only actionable when it traces down to the sessions that failed and the trajectories that broke. Leaders read the top of the stack, engineers manage the bottom, and the platform’s job is keeping the thread between them intact. An agent is improving when a change produces better task outcomes or greater efficiency without unacceptable regressions in policy compliance, latency, or cost. Session and trajectory evaluations help explain those changes by showing which tasks improved and how the agent completed them.

5 AI agent evaluation metrics to track in production

A working measurement system produces a small set of numbers a leader can track quarter over quarter, and each one resists a specific failure of the anecdote era:

• Task completion rate, by segment. The share of evaluated sessions that meet the workflow’s task-success criteria. Break results out by workflow, customer segment, and agent version, and state how much traffic was evaluated.
• Cost per completed task. Total model spend across successful and failed attempts, divided by completed tasks over the same period. Define which costs you include and track latency separately.
• Time from detection to verified fix. Elapsed time between discovering a failure and verifying that its fix works in production.
• Recurrence rate. The share of previously fixed failure patterns that reappear within a defined follow-up period.
• Discovered versus resolved. Newly identified failure patterns compared with patterns verified as resolved in the same period. Use this to track the issue backlog alongside task completion and recurrence.

Notice what’s absent: benchmark scores and thumbs-up rates. Both have uses, and neither tells a leader whether the agent does its job better this month than last.

Common mistakes when measuring AI agent performance

The measurement failures worth naming, since each one has fooled a real team:

• The changed ruler. Record which evaluator version produced each score. When the scoring criteria change, rerun the baseline and candidate with the updated evaluator before comparing results. Otherwise, an apparent improvement may reflect a different scoring standard.
• The Goodharted metric. Optimizing one score invites the agent to satisfy the score at the expense of the task, which is why the metric set stays layered, with cost and recurrence checking completion.
• The shifted traffic. Comparing this month to last month means little if the traffic mix changed, so comparisons need segment-level views and, for big changes, experiments against fixed datasets of real past failures.
• The survivor’s average. A healthy overall number can coexist with a collapsing segment, and the customers in that segment don’t experience the average.
• The passing output. Agents produce what researchers call corrupt success, where the answer passes while the process fails, so output-level scoring alone will overstate improvement in exactly the workflows with the most stakes.

How to review agent quality across releases

The cadence matters less than the questions, and a monthly agent quality review built on the metrics above stays short. The questions worth asking:

1. What’s the task completion trend, overall and by segment, and does anything below the top line explain the movement?
2. What did we discover this period that we weren’t testing for, and what did it cost before we caught it?
3. What did we fix, and did any of it come back?
4. What did we ship, and what’s the before-and-after on the fixed dataset it was tested against?
5. What’s cost per completed task doing, and where’s the spend concentrated?

Each answer should connect to supporting traces, evaluation results, or experiments. The review should make clear which changes improved task completion, which failures returned, and what the team needs to investigate next.

How to connect agent traces, evaluations, and experiments

Maintaining these metrics across releases requires connected traces, evaluation results, and experiment history.

• One record from development through production. Before-and-after comparisons require every call traced the same way across releases, on open standards that keep the record portable and the history intact.
• Evaluation at the session and trajectory level. Evaluate complete sessions for task success and agent trajectories for tool selection, retries, and policy compliance. Keep those scores linked to the underlying traces so you can investigate why a result changed.
• Versioned evaluators. Comparable scores over time require the judge under version control, so improvement in the number reflects improvement in the agent.
• Continuous discovery. An honest denominator requires something reviewing production traffic for failures nobody predefined, the way Signal groups related failures into prioritized issues, so the discovered-versus-resolved metric measures reality.
• Experiments against real failures. The before-and-after on every change needs fixed datasets built from actual production failures, with the same evaluators run in CI and against live traffic.
• A leader-readable view. The five review questions should be answerable from dashboards that product, support, and finance can read, while engineers work the layers underneath from the same record.

Start with one production workflow and one proposed change. Build an evaluation dataset from production traces, compare the existing agent and candidate on the same cases and criteria, and check whether the results hold after release. The next change should come with evidence of which tasks improved, what they cost, and whether previously fixed failures stayed fixed.

Related reading

• How to evaluate AI agents in production
• How to find and debug agent failures your evals are missing
• Session-level evaluations
• Agent trajectory evaluations
• Cost per successful task
• Writing LLM evals as CI tests
• LLM-as-a-Judge
• From production traces to better agents
• What is an agent observability platform?
• LLM evaluation

Measuring agent improvement FAQs

What metrics show an AI agent is improving?

Task completion rate by segment, cost per completed task, time from detection to verified fix, recurrence rate on fixed issues, and the balance of discovered versus resolved failure patterns. Together they show whether outcomes are rising, whether the improvement loop works, and whether the gains are real rather than an artifact of measurement.

Can we measure improvement with user feedback like thumbs ratings?

Feedback is a useful signal and a poor measurement system, since it samples a small, skewed slice of sessions and misses failures hidden behind plausible outputs. Treat it as one input to discovery rather than the metric of record, and rely on continuous evaluation for the denominator.

How do we compare agent quality before and after a model swap?

Run the candidate against a fixed dataset of real past failures with versioned evaluators, so the cases and the judge stay constant while the model changes, then confirm the offline result against live traffic with the same evaluators after release. Comparing raw production scores across the swap alone conflates the model change with traffic shifts.

How often should leaders review agent quality?

Monthly works for most teams, with alerting covering the gaps between reviews, and weekly during launches or major changes. The forcing function matters more than the frequency, since a standing review built on the five questions above keeps the team instrumenting for answers instead of assembling anecdotes.