Agent reliability is whether an agent completes the task it was given, correctly and within its budget, under ordinary conditions. No attacker, no unusual input, just the traffic you actually get. It is a property of the system, measured over many runs, and it is closer to task success rate than to uptime.
The underlying discipline is borrowed from site reliability engineering, but there is no standard metric set for agents the way there is for services. Nobody agrees on how to define task success across products, and the word is still doing double duty for correctness, consistency, and availability. The term is useful and you will have to say what you mean by it. This page describes the measurements that tend to matter.
The topic exists separately from service reliability because the metrics that work for services are nearly blind here. An agent can return 200 OK on every span, log no errors, finish in two seconds, and have failed the user completely.
Key takeaways
- Reliability is measured at the task level, not the request level. The question is whether the user’s job got done, not whether the service responded.
- The worst failures report success. No error, a confident final message, a session that ended normally, and nothing accomplished.
- Per-step accuracy compounds. High confidence at each step still produces a low completion rate across a long trajectory.
- Most of the fix lives in the harness: retries with idempotency, timeouts, step caps, stopping conditions, checkpoints, and a defined degraded path.
- Reliability is the property. Observability is how you find out whether you have it. Neither substitutes for the other.
Why service metrics miss it
Success codes are not success. The failure classes that reach users most often are the quiet ones: the agent loops between two tools until it hits a cap, ignores the document it retrieved, drops a constraint the user set ten turns ago, or updates one of three records and reports that it updated the records. Every span is healthy. Traditional systems either commit a transaction or roll it back; an agent can do three of five things, stop, leave state half-changed, and raise nothing. This is why a reliability program for agents starts with naming failure modes rather than with alert thresholds.
Nondeterminism is in the substrate. The same input can produce a different trajectory on the next run, so a single passing run proves very little and a single failure may not reproduce. Reliability here is a rate over repeated runs, which changes how you test and how you read a regression. Length compounds it: each additional step is another chance to pick the wrong tool, malform an argument, or misread a result, which is why long-running and multi-agent systems are hit hardest.
What to measure
Pick a small number and hold yourself to them.
- Task success rate. Defined per product, graded on the trajectory rather than the final message. The headline number, and the hardest one to define honestly.
- Tool call validity and success rate, per tool. Validation failures per tool per argument is cheap to compute from spans and one of the highest-yield numbers available.
- Recovery rate. Given a tool error, how often the agent recovers versus derails. Rarely measured, and systems differ enormously.
- No-progress and false completion rates. Sessions that hit a cap without an outcome, and sessions that claim work with no span to show for it.
- Cost and latency per completed task. Per task, not per call, so loops and retries land where they belong.
- Escalation and abandonment. Signals your product already emits, and the least gameable numbers on the list.
Then set a budget: a stated tolerance, such as a task success floor per segment, that a release has to clear. Without one, every regression conversation is about whether the number feels bad. Getting there depends on agent evaluation running against the trace rather than the response, because trajectory-level grading is what makes task success measurable at all.
The engineering that produces it
Reliability is mostly harness work, and almost all of it is unglamorous.
Retries with idempotency. Retry transient tool failures, and make the tools safe to call twice. An agent that retries a non-idempotent write is a reliability problem that creates a data problem.
Timeouts and caps everywhere. Step limits, token budgets, and wall-clock ceilings, with a defined behavior when one is hit. Hitting a cap should produce a clear outcome and a recorded reason, not a truncated message.
Explicit stopping conditions. A definition of done that does not depend on the model deciding it feels finished, plus verification for consequential claims: if the agent says it wrote the file, check for the write.
Checkpoints and a degraded path. Long-running work needs durable state so a resumed run does not redo completed side effects. And when the agent cannot finish, the good outcome is a clean handoff to a human with context attached, not a plausible answer covering for a failure.
Testing the unhappy path. Tool timeouts, empty results, malformed responses, and rate limits are the normal weather of a production agent, and the failures that traditional software tests miss are mostly in how the agent behaves when a dependency misbehaves.
Reliability and observability are not the same thing
Reliability is the property: the agent does what it should, at a rate you can state. It is produced by design decisions, controls, and tests.
Observability is the capability that lets you see whether the property holds: traces of every step with inputs and outputs, evals scored against those traces, and the ability to slice by segment. An agent observability platform is instrumentation and analysis, not a guarantee. Adding it does not make an agent reliable any more than a thermometer makes a room warm.
The dependency runs one way and it is strict. You cannot improve reliability you cannot measure, and you cannot measure agent reliability without trajectory-level visibility, because the failing step is usually not the step that raised the error. That is why teams build the observability first even though reliability is the goal.
FAQ
What is a good task success rate?
There is no general answer, and any number quoted without a task definition is meaningless. What matters is the floor you set for your own product, whether it holds across your riskiest segments, and whether it is trending the right way. A tightly scoped agent and an open-ended one are not comparable even inside the same company.
Is agent reliability the same as agent observability?
No. Reliability is whether the agent works. Observability is how you know. The two get conflated because the tooling conversation is about observability while the goal is reliability, and because instrumentation is the first thing you have to build to make progress on that goal.
Will a better model make my agent reliable?
It moves the failure mix rather than removing it. Stronger models select tools more accurately and malform fewer arguments. Dropped constraints, false completions, unbounded loops, and missing recovery paths are structural properties of the code around the model and survive the upgrade.
How does reliability relate to agent security?
Different conditions, overlapping controls. Reliability asks whether the system works when nothing is attacking it. Security asks what happens when something is. Permissions and limits improve both, but reliability is measured against your real traffic and security against inputs built to defeat you.
