1. Arize AX
  2. Phoenix OSS
  3. Docs
  4. Learn
  5. Company
  6. Pricing
Sign up Book a demo
Resources
Report

The agent reliability gap

Agent capability has advanced faster than the production systems around it. A strong model can still fail when an application supplies incomplete context, exposes the wrong tool, loses state, mishandles a dependency, or permits an action that violates policy. The final answer may still look plausible, which makes the system appear healthier than it is.

Based on 14 anonymized practitioner interviews, this paper gives AI engineers, product teams, and technical leaders a practical framework for treating reliability as a property of the full agent system.

Download the report to learn how to:

  • Classify failures across six recurring domains and capture the evidence each one requires.
  • Reconstruct long-running, multi-step behavior with a minimum production trace schema.
  • Turn production failures into product-specific evals, datasets, and regression cases.
  • Apply runtime controls for identity, permissions, approvals, audit, rollback, and deterministic verification.
  • Run a 30-day reliability sprint that connects engineering, product, domain, platform, and security teams.
The production agent reliability stack
  1. 01 RUNTIME OBSERVABILITY AND GOVERNANCE
  2. 02 MODEL CHOICE
  3. 03 ORCHESTRATION, ROUTING, AND VERIFICATION
  4. 04 TOOLS AND DATA
  5. 05 CONTEXT AND MEMORY
  6. 06 USER OBJECTIVE AND PRODUCT EXPERIENCE

Get the report

Download the full report with the six-domain failure taxonomy, minimum production trace schema, runtime governance controls, shared operating model, and 30-day reliability sprint.

THE RELIABILITY GAP

A workable output can hide a broken run

Output quality remains essential because users care whether the answer or action met their needs. It cannot explain the execution path on its own. A model may compensate for missing context, retry around a failing dependency, or choose an expensive path and still satisfy the user.

This gap separates an acceptable output from dependable execution. Teams need enough evidence to reconstruct the run and identify whether the failure came from reasoning, context, tools, state, infrastructure, evaluation, or policy.

THE CENTRAL OPERATING PRINCIPLE

Every consequential agent run should produce enough evidence to explain what the agent attempted, why it selected each action, which resources it used, how well it met the user objective, and whether its behavior stayed within policy.

A PRACTICAL TAXONOMY

Where agent reliability breaks

Reliability improves faster when teams classify the failure before choosing a fix. The paper groups recurring failures into six domains, each with distinct evidence and a different owner.

Context and intent

Ambiguous requests, missing business context, stale retrieval, or weak instructions. Inspect inputs, clarifications, context versions, and memory operations.

Tools and access

Unavailable tools, malformed schemas, unsuitable interfaces, or wrong credentials. Capture selection, arguments, outputs, permissions, errors, and retries.

Orchestration and state

Loops, premature stops, retry storms, compaction loss, or incorrect routing. Trace decisions, state changes, budgets, handoffs, and stop reasons.

Data and infrastructure

Partial results, timeouts, stale data, dependency latency, or load failures. Connect agent spans with queries, timestamps, and service telemetry.

Evaluation and product fit

Generic scores, rubric drift, or weak alignment with the user outcome. Review representative traces, product metrics, and evaluator versions.

Safety and governance

Unauthorized actions, boundary crossing, excessive scope, or hidden side effects. Preserve policy decisions, approvals, audit records, and rollback steps.

“The agent decided to delete all the tests because then none of them would fail.”
Michael Grinich Founder, WorkOS

At WorkOS, an agent was asked to make sure all tests passed and then ended up deleting all the tests. While it worked against the goal itself, it also destroyed the mechanism that made the goal useful in the first place.

A successful outcome can still conceal a control failure. An agent can cross a security boundary, exceed its scope, or create an unacceptable side effect while appearing to complete the task.

HOW TO APPLY IT

A framework you can put into production

The report turns the research into six operating tools that engineering, product, domain, platform, and security teams can use together.

Six-domain evidence matrix

Match each recurring failure class to the signals, evidence, and owner needed to diagnose it.

Minimum production trace schema

Record input, versions, context, decisions, execution, economics, governance, and outcome for every consequential run.

Human-guided improvement loop

Instrument, inspect, classify, define evaluators, review with domain experts, then replay and monitor.

Runtime governance controls

Apply agent identity, least privilege, risk-aware approval, audit and rollback, and deterministic verification.

30-day reliability sprint

Trace one high-value workflow, build the failure map, create five to ten evals, then replay and release.

Methodology

About the research

Arize AI conducted 14 anonymized interviews in 2026 with practitioners from frontier model labs, enterprise AI platforms, cloud infrastructure, developer tools, consumer applications, and voice AI. Recurring themes are reported in aggregate.

Keep reading

Recommended resources

See all resources
Blog

From Signal to PR: What if your agents got better every time they failed?

Signal, a managed agent built into Arize AX, continuously reviews production traces, surfaces ranked issues with evidence and…

Read the post
Blog

How Signal found two hidden retry loops in our production agent Alyx

We ran Signal on Alyx, the AI engineering agent built into Arize AX. It surfaced a duplicate task-state…

Read the post
Case Studies

How Tripadvisor is building the AI product development lifecycle for agentic travel

Tripadvisor VP of Data and AI Rahul Todkar on building a production AI lifecycle for traditional ML and…

Read the story
Guide

AI model lifecycle management: 7 stages, controls, and tools

The seven stages of AI model lifecycle management, what to version at each gate, which tools own which…

Read the guide

Build a feedback loop for reliable agents

Download the full paper for the failure taxonomy, minimum trace schema, runtime governance controls, shared ownership model, and 30-day implementation plan.

Download the report