AI agent guardrails vs. evals: How to build more reliable agent systems

The voice agent knew what to say. But then it began saying three things at once.

One response started, another cut in, and a third arrived before either of the first two had finished. The voices overlapped until the answer became impossible to follow.

Laurie Voss, head of developer relations at Arize, watched the failure unfold during a live demonstration. The language model had understood the request, but the system had simply allowed several response paths to proceed simultaneously.

“There was nothing wrong with the model,” Voss said. “There was definitely something wrong with the harness.”

The incident captures a problem that will become harder to ignore as AI agents run longer and act with less supervision. Model capability can carry an agent surprisingly far, but reliable behavior depends on the system that determines which tools it can use, what actions it can take, how its work is evaluated, and what happens after something goes wrong.

That system needs both guardrails and evals. The terms often appear together, yet they solve different engineering problems.

An hour-long agent run can fail in more places than a chatbot

Early AI agents required almost constant supervision. A developer might let one run for a few minutes, watch what happened, and intervene when it wandered off course.

That autonomy window is expanding. Voss hears developers describe agents that can work for several hours, sometimes beginning with a product requirements document and returning with a substantial implementation. Whether every two-day coding run succeeds is less important than the direction of travel. Agents are being given more time, more tools, and more authority.

A conventional chatbot interaction has a narrow failure surface. It receives a message and generates a response. A long-running agent may retrieve information, call several tools, modify files, retry failed operations, manage state, delegate work, and decide when the task is complete.

Each new step creates another place where the system can deviate from the intended path:

• A tool call can use the wrong arguments.
• A retry loop can consume excessive time or tokens.
• Two branches can attempt conflicting actions.
• The agent can carry stale context into a later decision.
• A valid intermediate action can produce a poor final outcome.
• A useful final answer can conceal a wasteful or unsafe trajectory.

For developers, longer trajectories create an orchestration and observability problem. For product managers, greater autonomy changes the product contract. The team must decide how much authority the agent receives, when it needs approval, how it should recover, and what the user sees when the workflow leaves its expected path.

Evals and guardrails solve different problems

Voss draws a useful distinction between evals and guardrails.

“The evals say what’s good and what’s bad, but the guardrails are enforced at the code level,” he explains.

An eval judges behavior while a guardrail constrains behavior. Here’s an easy way to break this down:

The voice agent needed a guardrail that permitted only one active answer. An eval could detect the overlapping speech after the run, but detection alone would not prevent the behavior from recurring.

Reliable systems translate product requirements into executable constraints. “Stay focused” is difficult to enforce. “Permit one active voice response at a time” can become a testable rule.

A simplified policy might look like this:

# Illustrative pseudocode

policy = AgentPolicy(
allowed_tools={“knowledge_search”, “calendar_lookup”},
max_parallel_responses=1,
max_retries=2,
approval_required={“send_email”, “make_purchase”},
)

Prompt instructions still matter because they help shape the agent’s plan. Code-level controls provide a firmer boundary when the action carries cost, risk, or an irreversible consequence.

Useful guardrails can cover:

• which tools the agent may invoke;
• which credentials each tool receives;
• how many actions may run concurrently;
• how long a workflow may continue;
• how many times a failed step may retry;
• which actions require human approval;
• when the agent should stop, escalate, or ask for clarification.

A high-quality final answer does not prove that the agent followed an acceptable path. Teams need to evaluate the outcome while also enforcing the boundaries around the process.

Your AI judge may be grading yesterday’s world

Guardrails can fail and so can evaluators.

Voss uses a financial-analysis agent in her tutorials. The agent researches recent events and produces a report. A generic correctness evaluator then grades that report using its own internal knowledge.

The result is a predictable mismatch. The agent cites information from the current day, while the evaluator behaves as though the newer events have not happened. It marks the report incorrect every time because its frame of reference is stale.

The evaluator appears authoritative because it produces a score. That score still reflects the context available to the judge.

An evaluator may need access to:

• the current date and time;
• the sources retrieved during the run;
• the user’s actual goal;
• the policies governing the product;
• the tools available to the agent;
• the complete trajectory;
• the product’s definition of a successful outcome.

“Correctness” is rarely universal. A current financial report, a customer-support response, and a code migration each require different evidence. A generic judge may evaluate fluency while missing the business rule that determines whether the answer is usable.

Developers should test evaluators with the same skepticism they apply to the application. Product managers should define what evidence a judge needs before its score can influence a release decision.

A failed run can become the next engineering brief

The most useful eval output may be an explanation rather than a score.

In her own side projects, Voss runs evals that describe what went wrong. Her development environment pulls those explanations from Arize and gives them to the coding agent responsible for the next iteration.

The instruction is blunt: “Here are the explanations of all the things that went wrong in your last run. Fix yourself.”

The workflow creates a bounded form of self-improvement:

Agent run
    ↓
Traces and outputs
    ↓
Evals generate scores and explanations
    ↓
Failure explanations return to the IDE
    ↓
Coding agent proposes changes
    ↓
Tests and evals run again

This loop does not require an agent to redesign and deploy itself without oversight. Evaluation feedback becomes a structured development task, which a coding agent can act on before the next review.

The distinction matters. A pass/fail result can tell a team that quality declined. An explanation can guide a specific change to:

• a prompt;
• a tool description;
• a retrieval query;
• a permission boundary;
• a retry policy;
• a piece of application code;
• a regression test.

The workflow also changes how teams think about production failures. A bad run no longer has to remain an isolated incident in a trace viewer. It can become evidence for the next version of the system.

Product requirements need operational counterparts

Product teams often describe agent behavior using language that sounds clear during planning but becomes ambiguous during implementation.

Consider a voice-booking agent. A requirement might say: The agent should stay focused on completing the booking.

That sentence leaves several questions unresolved. Can the agent react to background speech? Should it answer unrelated questions during the booking? When has the task ended? What happens when the user changes direction halfway through?

A more operational definition might specify that the agent must:

1. act only on speech directed toward the booking workflow;
2. maintain one active response at a time;
3. ignore unrelated background conversation;
4. request clarification when the destination is ambiguous;
5. stop after confirmation or transfer control to a human.

These conditions can become guardrails, eval criteria, or both.

The product manager defines the intended experience. The developer converts that intent into system behavior. The eval reveals whether the experience occurred, while the guardrail prevents actions that should never happen.

Five questions to ask before increasing agent autonomy

Before allowing an agent to run longer or act with less supervision, the team should be able to answer five questions.

1. What can the agent do without approval?

List the permitted actions explicitly. Separate reversible work from actions that spend money, modify production data, communicate externally, or create legal obligations.

2. Which boundaries are enforced in code?

Prompt instructions express desired behavior. Consequential restrictions should also have executable enforcement.

3. What context does the evaluator need?

A judge cannot assess current information, domain policy, or tool use unless the relevant evidence is available during evaluation.

4. Can the team reconstruct the trajectory?

A final answer may hide retries, unnecessary tool calls, conflicting branches, or policy violations. Traces should expose the steps that produced the result.

5. How does failure information reach the next version?

Decide whether failures become tickets, regression tests, updated evals, prompt changes, guardrail changes, or structured feedback for a coding agent.

Without that last connection, observability remains retrospective. The team learns what happened but does not systematically change what will happen next.

The AI engineer becomes a systems engineer

Voss defines an AI engineer as someone who brings substantial expertise from engineering or another domain, then uses AI effectively as another tool.

That definition places judgment above novelty. The work requires more than knowing which model or framework was released last week. AI engineers increasingly need to reason across several layers at once:

• model behavior
• product intent
• orchestration
• permissions
• guardrails
• evaluation design
• observability
• feedback and iteration

The language model may generate the answer, choose a tool, or propose a code change. The engineer builds the conditions under which those actions become dependable.

As agents operate for longer periods, reliability will depend on systems that constrain what they can do, evaluate what they actually did, and carry useful evidence from one run into the next.

Case in point via our original example: the model knew how to answer the question in the voice demo, but the engineering challenge was making sure only one answer reached the user (and not three all at the same time).