Context and intent
Ambiguous requests, missing business context, stale retrieval, or weak instructions. Inspect inputs, clarifications, context versions, and memory operations.
Report
The agent reliability gap
Agent capability has advanced faster than the production systems around it. A strong model can still fail when an application supplies incomplete context, exposes the wrong tool, loses state, mishandles a dependency, or permits an action that violates policy. The final answer may still look plausible, which makes the system appear healthier than it is.
Based on 14 anonymized practitioner interviews, this paper gives AI engineers, product teams, and technical leaders a practical framework for treating reliability as a property of the full agent system.
Download the report to learn how to:
- Classify failures across six recurring domains and capture the evidence each one requires.
- Reconstruct long-running, multi-step behavior with a minimum production trace schema.
- Turn production failures into product-specific evals, datasets, and regression cases.
- Apply runtime controls for identity, permissions, approvals, audit, rollback, and deterministic verification.
- Run a 30-day reliability sprint that connects engineering, product, domain, platform, and security teams.
Arize AI
THE AGENT RELIABILITY GAP
How observability addresses the system failures that stronger models can hide.
Framework The production agent reliability stack
- 1 RUNTIME OBSERVABILITY AND GOVERNANCE
- 2 MODEL CHOICE
- 3 ORCHESTRATION, ROUTING, AND VERIFICATION
- 4 TOOLS AND DATA
- 5 CONTEXT AND MEMORY
- 6 USER OBJECTIVE AND PRODUCT EXPERIENCE
THE RELIABILITY GAP
A workable output can hide a broken run
Output quality remains essential because users care whether the answer or action met their needs. It cannot explain the execution path on its own. A model may compensate for missing context, retry around a failing dependency, or choose an expensive path and still satisfy the user.
This gap separates an acceptable output from dependable execution. Teams need enough evidence to reconstruct the run and identify whether the failure came from reasoning, context, tools, state, infrastructure, evaluation, or policy.
THE CENTRAL OPERATING PRINCIPLE
Every consequential agent run should produce enough evidence to explain what the agent attempted, why it selected each action, which resources it used, how well it met the user objective, and whether its behavior stayed within policy.
A PRACTICAL TAXONOMY
Where agent reliability breaks
Reliability improves faster when teams classify the failure before choosing a fix. The paper groups recurring failures into six domains, each with distinct evidence and a different owner.
Tools and access
Unavailable tools, malformed schemas, unsuitable interfaces, or wrong credentials. Capture selection, arguments, outputs, permissions, errors, and retries.
Orchestration and state
Loops, premature stops, retry storms, compaction loss, or incorrect routing. Trace decisions, state changes, budgets, handoffs, and stop reasons.
Data and infrastructure
Partial results, timeouts, stale data, dependency latency, or load failures. Connect agent spans with queries, timestamps, and service telemetry.
Evaluation and product fit
Generic scores, rubric drift, or weak alignment with the user outcome. Review representative traces, product metrics, and evaluator versions.
Safety and governance
Unauthorized actions, boundary crossing, excessive scope, or hidden side effects. Preserve policy decisions, approvals, audit records, and rollback steps.
“The agent decided to delete all the tests because then none of them would fail.”Michael Grinich Founder, WorkOS
At WorkOS, an agent was asked to make sure all tests passed and then ended up deleting all the tests. While it worked against the goal itself, it also destroyed the mechanism that made the goal useful in the first place.
A successful outcome can still conceal a control failure. An agent can cross a security boundary, exceed its scope, or create an unacceptable side effect while appearing to complete the task.
HOW TO APPLY IT
A framework you can put into production
The report turns the research into six operating tools that engineering, product, domain, platform, and security teams can use together.
Production reliability stack
Connect the user outcome to context and memory, tools and data, orchestration, model behavior, and runtime governance.
Six-domain evidence matrix
Match each recurring failure class to the signals, evidence, and owner needed to diagnose it.
Minimum production trace schema
Record input, versions, context, decisions, execution, economics, governance, and outcome for every consequential run.
Human-guided improvement loop
Instrument, inspect, classify, define evaluators, review with domain experts, then replay and monitor.
Runtime governance controls
Apply agent identity, least privilege, risk-aware approval, audit and rollback, and deterministic verification.
30-day reliability sprint
Trace one high-value workflow, build the failure map, create five to ten evals, then replay and release.
Methodology
About the research
Arize AI conducted 14 anonymized interviews in 2026 with practitioners from frontier model labs, enterprise AI platforms, cloud infrastructure, developer tools, consumer applications, and voice AI. Recurring themes are reported in aggregate.
Build a feedback loop for reliable agents
Download the full paper for the failure taxonomy, minimum trace schema, runtime governance controls, shared ownership model, and 30-day implementation plan.
Keep reading
See all resources
Recommended resources
Blog
From Signal to PR: What if your agents got better every time they failed?
Signal, a managed agent built into Arize AX, continuously reviews production traces, surfaces ranked issues with evidence and…
Read the post
Blog
How Signal found two hidden retry loops in our production agent Alyx
We ran Signal on Alyx, the AI engineering agent built into Arize AX. It surfaced a duplicate task-state…
Read the post
Case Studies
How Tripadvisor is building the AI product development lifecycle for agentic travel
Tripadvisor VP of Data and AI Rahul Todkar on building a production AI lifecycle for traditional ML and…
Read the storyDon’t ship vibes.
Arize gives AI teams observability and evals to understand and improve agent performance.
