AI agent testing: 7 failures traditional software tests miss

An agent books a flight and reports, “Your flight has been booked.”

The response is clear. The request returned successfully. No exception was thrown.

There is only one problem: no reservation exists.

This is the coverage gap that appears when teams apply conventional software testing patterns to AI agents. Unit, integration, and API tests can verify that code executed, schemas were valid, and individual services returned successfully. Those checks cannot always prove that an agent understood the request, selected the right tools, changed the correct application state, and accurately reported the result.

This is why AI agents need agent evals. Evals extend the existing test stack by measuring task outcomes, tool use, multi-step behavior, and response accuracy together.

This article examines seven agent failures that request-level and output-level tests commonly miss. For the full process of building datasets, selecting evaluators, running experiments, and turning production failures into regression tests, see the guide to evaluating AI agents in production.

Why successful requests can hide failed agent tasks

For a conventional API, the returned value often provides a reasonable representation of what the system did. A successful response from a reservation endpoint may correspond directly to a confirmed row in a database.

An agent introduces another layer between the user and the underlying systems. Before generating its response, the agent may:

• Interpret the user’s request
• Retrieve information from multiple sources
• Select between available tools
• Generate and validate tool arguments
• Perform several dependent actions
• Read or modify application state
• Carry context across multiple turns

The final response is generated from that process. It is evidence about what happened, but it is not proof that the task succeeded.

Consider a flight-booking agent. A successful task might require all of the following conditions:

reservation.status == "confirmed"
reservation.destination == requested_destination
reservation.departure_date == requested_date
payment.amount == quoted_total
confirmation.reservation_id == reservation.id

A conventional test might confirm that the request returned an HTTP 200 response, produced valid JSON, and did not throw an exception. An evaluator that reads only the final confirmation might also grade “Your flight has been booked” as correct.

Neither check establishes that:

• The booking tool was called
• The requested flight was selected
• The correct date was used
• Payment completed exactly once
• A confirmed reservation exists
• The confirmation number matches that reservation

The response is one part of the agent’s behavior. The environment state and the steps that produced it also need to be tested.

None of this makes deterministic testing obsolete. Any requirement that can be expressed reliably as an assertion should remain an assertion. The gap appears when a test suite validates the agent’s components without validating the agent’s end-to-end behavior.

7 AI agent failures traditional tests miss

1. The agent reports success without completing the task

The most direct failure occurs when the agent claims that an action succeeded even though the expected state change never happened.

A support agent may say that it closed a ticket while leaving the ticket open. A coding agent may report that it fixed a bug even though the test suite still fails. A scheduling agent may confirm a meeting without creating the calendar event.

From the application’s perspective, several components may have worked correctly:

• The model generated a valid tool call.
• The tool returned without an exception.
• The agent produced a well-formed final response.

The user’s task still failed.

What catches it: Verify the resulting environment state directly, then compare the final response with that state. For the flight-booking example, the evaluator should inspect the reservation system rather than infer success from the agent’s wording.

2. The agent selects the wrong tool or passes the wrong arguments

Tool endpoints can work exactly as designed while the agent uses them incorrectly.

An agent might:

• Call search_flights when the task requires create_reservation
• Use an account ID from an earlier conversation
• Pass the arrival date as the departure date
• Omit a required argument
• Repeat a write operation after a slow response
• Select a read-only tool for a task that requires a state change

A unit test for each tool will not expose this failure. The tools themselves may be healthy. The error lies in the agent’s decision about which tool to use and how to use it.

What catches it: Evaluate tool selection, argument validity, account and permission boundaries, schema compliance, and duplicate actions. These checks can often be implemented as inexpensive code-based evaluators. See the guide to evaluating tool-calling agents for a deeper treatment of tool-level failure modes.

3. The agent skips a required step in a multi-step workflow

Many agent tasks succeed only when several actions occur in the correct dependency order.

Booking a flight may require the agent to:

1. Find an available itinerary.
2. Confirm the price and constraints.
3. Create the reservation.
4. Process payment.
5. Return the reservation identifier.

An agent could complete steps one, two, and three, then generate a confirmation without processing payment. Every tool call that did occur may have succeeded, but the workflow remains incomplete.

This failure is common in multi-step agent workflows because component-level tests usually inspect actions in isolation. They do not necessarily verify that all required actions occurred before the agent declared success.

What catches it: Define the required state transitions or actions for the task. Evaluate whether the trace contains those requirements without forcing one exact sequence when several valid paths exist.

4. An early error changes everything the agent does next

In a conventional request pipeline, a failed operation may produce an exception and stop execution. Agents often continue after a bad decision.

A misrouted tool call at step two can change the context available at step three. That altered context affects the next tool selection, which may modify the environment again. By the time the agent produces its final response, the original mistake has propagated across the entire run.

A 2025 survey of agent evaluation describes this cascading behavior in dynamic environments: an early action changes what the agent observes downstream, allowing one error to compound into task failure.

An output evaluator may identify that the final answer is wrong, but it cannot explain where the run first diverged or why.

What catches it: Evaluate the complete trace and inspect the first action that violated the task contract. Trace-level evaluation makes it possible to separate the root error from the downstream symptoms it created.

5. The agent reaches the correct outcome through a risky or wasteful path

A correct final state does not guarantee acceptable behavior.

An agent might eventually complete the task after:

• Calling the same tool repeatedly
• Ignoring a tool error and continuing with stale data
• Accessing information it did not need
• Taking an action before gathering required information
• Attempting a prohibited operation before recovering
• Generating excessive latency or token cost

A test that checks only the final database state may pass. From a reliability, security, or cost perspective, the run should still fail.

At the same time, an evaluator should not require one exact sequence unless that sequence is part of the product contract. Agents may discover several valid ways to complete a task.

What catches it: Evaluate required actions, prohibited actions, unnecessary repetition, ignored errors, and task-specific efficiency constraints. Trajectory evals are designed to assess the ordered sequence of observable actions rather than only the final output.

6. The agent loses context across multiple turns

Some failures remain invisible when each conversation turn is evaluated separately.

An agent may answer an individual request correctly while:

• Forgetting a user preference stated earlier
• Acting on an outdated account or record
• Overwriting useful memory
• Contradicting a previous commitment
• Losing a permission or safety constraint
• Repeating an action completed in an earlier turn

For example, a travel agent might remember the destination but forget that the user rejected overnight flights. The final itinerary could look reasonable when viewed alone, even though it violates the established session context.

What catches it: Match the evaluation boundary to the task boundary. Use a span-level check for one operation, a trace-level eval for one complete agent run, and a session-level eval when success depends on information or actions across several turns.

7. The agent passes once but fails unpredictably across repeated runs

Agent behavior can vary even when the input and application code remain unchanged.

One run may select the correct tool and complete the task. The next may choose a different plan, omit a step, or misinterpret the same tool response. A single successful test run therefore provides limited evidence about production reliability.

This matters most for high-frequency or high-impact tasks. A workflow that succeeds nine times out of ten may look strong in a demo while still producing an unacceptable number of failures at production scale.

What catches it: Run important cases repeatedly and track:

• Task success rate
• Failure rate by failure mode
• Variance between trials
• Latency and cost distributions
• Performance by task category

Use pass@k when the product intentionally allows several attempts and only one needs to succeed. For an agent expected to complete a task correctly on its first attempt, first-attempt success rate is the more representative metric.

What agent evals add to the test stack

Agent evals do not replace unit, integration, API, or end-to-end tests. They add coverage for behavior that is distributed across model decisions, tool calls, state changes, and conversation turns.

A complete agent test stack may include:

Deterministic checks remain the best option for deterministic contracts, including:

• JSON schema validation
• Required tool arguments
• Status codes
• Database invariants
• Permission checks
• Exact calculations
• Duplicate-action detection
• Latency and token limits

Semantic evaluators are useful when correctness cannot be expressed reliably as code. An LLM-as-a-judge evaluator can assess whether a plan was sensible, whether the agent followed the user’s instructions, whether it preserved relevant context, or whether the final explanation accurately describes the work completed.

Human review remains important for calibrating subjective evaluators and resolving ambiguous, high-impact cases.

How to choose between a traditional test and an agent eval

For each requirement, ask what evidence would prove that the behavior was acceptable.

1. Can the requirement be expressed as an exact invariant? Use a code-based assertion. Examples include schema validity, status values, account IDs, permission rules, exact calculations, and database state.
2. Does correctness depend on meaning or judgment? Use a semantic evaluator. Examples include planning quality, instruction adherence, response usefulness, and whether the explanation matches a complex result.
3. Does success depend on the actions taken along the way? Evaluate the trace or trajectory. Check for required actions, prohibited actions, ignored errors, and unnecessary repetition.
4. Does the task span several conversation turns? Evaluate the complete session rather than grading each response independently.
5. Could the behavior vary between runs? Run repeated trials and compare success rates and failure modes against a baseline.

Use the cheapest reliable evaluator for each requirement. A deterministic assertion is usually preferable to an LLM judge when both can enforce the same contract. Semantic evaluators should cover requirements that cannot be reduced to exact code without losing important context.

Agent evals create a better definition of “working”

An agent is not successful simply because it returned a response.

It succeeds when it completes the requested task, changes the correct system state, follows the required constraints, and accurately reports what happened. For non-deterministic systems, it must also do so consistently enough for the product’s risk tolerance.

A practical starting point is one important workflow:

1. Write down what must be true in the environment after the agent finishes.
2. Identify actions the agent must take and actions it must never take.
3. Check that the final response agrees with the resulting state.
4. Run those checks against representative traces and repeated trials.
5. Save meaningful failures as future regression cases.

This creates a stronger acceptance criterion than “the request completed” or “the answer looked correct.”

Arize Phoenix supports tracing model calls, retrieval, tool use, and application logic, along with datasets and code-based or LLM-based evaluations. Arize AX can apply evaluators to production traces and sessions so that teams can identify new failure patterns and turn them into test cases.

For the complete engineering workflow, including dataset creation, evaluator selection, experimentation, CI, and production monitoring, read How to Evaluate AI Agents: A Production Workflow. For the underlying measurement model, see the agent-native evaluation framework.

FAQs

Do agent evals replace unit and integration tests?

No. Unit and integration tests remain the best tools for deterministic component behavior and application contracts.

Agent evals add coverage for task-level outcomes, tool decisions, trajectories, semantic behavior, and multi-turn context. A production agent typically needs both conventional tests and agent-specific evaluations.

How do agent evals differ from standard LLM evals?

Standard LLM evals often score one generated response for properties such as correctness, relevance, or style.

Agent evals may also inspect tool calls, arguments, application state, multi-step trajectories, and behavior across several turns. A final response can pass a text-quality evaluator while the underlying task still fails.

Can I evaluate an agent using only its final response?

A response-only evaluator may be sufficient for a task that produces text without interacting with tools or external state.

Once an agent performs actions, the final response should be evaluated alongside the tool calls, resulting environment state, and relevant trajectory. Otherwise, the evaluator may reward a convincing description of work that never happened.

What agent behavior should I evaluate first?

Start with one high-value workflow and the failure that would matter most to the user or business.

Define the environment state that proves success, then add checks for the most important tool, permission, or trajectory constraint. A narrow evaluator tied to a real failure is more useful than a broad score with no clear relationship to task success.

When should I use code-based evaluators instead of LLM-as-a-judge?

Use code-based evaluators whenever a requirement can be expressed deterministically. Examples include schemas, exact values, tool arguments, database state, permission rules, and duplicate actions.

Use LLM-as-a-judge for semantic properties such as planning quality, instruction adherence, context use, and whether a response accurately summarizes a complex result.

How should I handle non-determinism in agent testing?

Run multiple trials for important examples and compare success rates against a baseline. Track individual failure modes rather than relying only on one aggregate score.

Avoid requiring an exact trajectory when several valid trajectories exist. Reserve strict release blocking for stable, well-calibrated evaluators tied to high-confidence requirements.