How to evaluate AI agents

What is an AI agent?

Quick answer: an AI agent is a software system that uses a model to decide what action to take next, executes that action through tools, observes the result, and repeats the process until it reaches a goal or a stop condition.

An agent usually combines six elements:

• A task or goal that defines the desired outcome
• A model that interprets context and makes one or more decisions
• Instructions and policies that describe expected behavior and constraints
• Tools that let the system retrieve information or act on an environment
• State or context that carries relevant information across steps
• A control loop that decides whether to continue, finish, retry, or escalate

Multi-step processing alone does not make a system agentic. A fixed pipeline can call an LLM five times while application code determines every transition. An agent gives the model some authority over the next step, tool, subtask, or stopping decision.

The interface also does not determine whether a system is an agent. Agents can begin from a chat message, API request, scheduled job, queue event, monitoring alert, or another agent. Some communicate with users continuously. Others run in the background and return an artifact, state change, or verified result.

The term has no single industry-wide boundary. Anthropic separates workflows, where LLMs and tools are orchestrated through predefined code paths, from agents, where models dynamically direct their own processes and tool usage. OpenAI describes single-agent systems as a model-and-tool loop that runs until an exit condition, typically a final output, a structured result, an error, or a maximum turn count. These definitions point to the same architectural question: how much control does the model have over execution? See Building effective agents and A practical guide to building agents.

AI agents vs. agentic AI

The two terms are often used interchangeably, and the distinction is mostly one of scope rather than architecture. “AI agent” names a specific system: this service, with these tools, pursuing this task. “Agentic AI” is a category label for the broader class of systems that plan and act with some autonomy, including agentic workflows where a model makes only bounded decisions inside code-defined paths.

Nothing in either term tells you how much authority the system actually has. That is why the two-axis framing below is more useful than the vocabulary.

What makes a system agentic?

A useful way to describe agentic systems is with two independent axes.

1. Decision authority

Decision authority describes who selects the next step.

• Code-directed: application logic determines the sequence, branches, and termination.
• Model-assisted: the model makes bounded choices inside a larger deterministic workflow.
• Model-directed: the model selects tools, delegates subtasks, revises its plan, or decides when to finish within defined limits.

2. Action authority

Action authority describes what the system can do.

• Answer-only: generate text, code, classifications, or structured output.
• Read: search documents, query databases, inspect files, or retrieve external state.
• Write: update records, create files, send messages, execute code, or change another system.
• High-impact action: approve transactions, modify production infrastructure, change access, or perform another consequential operation.

This framing matters because autonomy is a property of the complete system, not a property of the model alone. The same model can power a read-only assistant, a tightly constrained workflow, or an agent that can modify production systems. Tool access, authorization, orchestration, approval gates, and stop conditions determine the operating envelope. We watched this play out directly when we ran seven models under a single agent harness: the harness shaped behavior at least as much as the model choice did.

The agentic spectrum

Teams can move a system left or right on this spectrum without rebuilding it from scratch. Reducing tool scope, replacing a model router with code, requiring approval, or making termination explicit lowers autonomy. Adding model-directed planning, write tools, delegation, or long-running execution increases it.

AI agents vs. chatbots, assistants, RAG, and workflows

These terms describe overlapping systems, so product labels are often less useful than architecture.

A chatbot can be an agent when it can decide and act. An agent can have no chat interface at all. A RAG system becomes agentic RAG when the model decides whether to retrieve, formulates its own queries, evaluates what it found, and repeats the search. A workflow becomes more agentic as models gain authority over routing, tool selection, planning, and termination.

How do AI agents work?

Most agents implement a feedback loop. The exact architecture varies, but the runtime repeatedly connects a model decision to an observable result. Our masterclass on agent workflows and architectures walks through the same loop with running code.

1. Receive a task and establish a task contract

The system records the user’s goal, available context, allowed actions, required approvals, budget, and success criteria. A vague request such as “fix the account” should become a bounded contract that identifies the account, the permitted changes, and the evidence that would confirm completion.

2. Assemble context

The runtime builds the information the model needs for the current decision. Context may include:

• System and developer instructions
• The user’s request
• Conversation or session state
• Retrieved documents
• Previous tool results
• Current environment state
• Available tools and their schemas
• Policies, permissions, and remaining limits

Context should be selected for the current step rather than accumulated without control. Long histories, stale tool outputs, and irrelevant retrieved text make decisions harder to reproduce and debug. Context assembly is where most long-running agents quietly degrade, which is why managing memory beyond the context window is a design problem rather than a tuning problem.

3. Decide or plan

The model proposes a next action. Depending on the architecture, it may:

• Return a final answer
• Select a tool
• Choose a branch
• Decompose the task
• Delegate a subtask
• Ask the user for missing information
• Request human approval
• Stop because the task cannot be completed safely

Some systems create an explicit plan. Others decide one step at a time. Planning is useful when the task has dependencies, but a detailed plan can become stale as soon as the environment changes. Many reliable systems combine a lightweight plan with repeated replanning from observed results. We wrote up the version of this that survived contact with production in how to build planning into your agent.

4. Authorize and execute the action

The runtime validates the proposed action before execution. Validation may include schema checks, authentication, user-specific authorization, policy rules, rate limits, risk classification, and human approval.

The tool then interacts with the environment. It might query a database, search the web, edit a file, run code, call an internal API, send a message, or invoke another agent.

5. Observe the result

The tool returns structured evidence: data, an error, a changed resource, a test result, or another observable state. The agent should use this evidence rather than assume the action succeeded.

6. Update state

The runtime records what happened, what changed, what remains unresolved, and which limits have been consumed. For durable agents, a checkpoint can preserve enough execution state to resume after a pause, restart, timeout, or approval.

7. Verify, continue, stop, or escalate

The system checks whether the task has actually succeeded. A support agent should verify that a refund exists in the payment system. A coding agent should run relevant tests. A data agent should reconcile its result against source data. The final response is evidence only when the task itself is to produce a response.

A run should also stop when it reaches a maximum number of steps, time limit, token or cost budget, repeated failure threshold, policy boundary, or condition requiring human judgment.

A minimal agent loop

def run_agent(user_request, user, tools, policies):
    state = initialize_run(
        task=user_request,
        user=user,
        limits={"max_steps": 12, "max_cost_usd": 2.00, "timeout_s": 120},
    )

    while not state.is_terminal():
        context = build_context(
            task=state.task,
            state=state,
            tools=tools.allowed_for(state.user, state.task),
            policies=policies,
        )

        decision = model.choose_next_action(context)
        authorization = authorize(decision, state, policies)

        if authorization.denied:
            state.record_denial(decision, authorization.reason)
            if state.consecutive_denials >= 2:
                state.stop(reason="policy_denied")
            continue

        if authorization.requires_human_approval:
            approval = request_approval(decision, authorization.bound_parameters)
            if not approval.granted:
                state.stop(reason="approval_denied")
                break

        try:
            observation = execute(decision, authorization)
        except TransientToolError as err:
            observation = Observation.retryable(err)
        except TerminalToolError as err:
            observation = Observation.terminal(err)

        state.record(decision=decision, observation=observation)
        emit_span(decision, observation, state)

        if verify_success(state):
            state.complete()
        elif limits_reached(state):
            state.stop(reason="limit_reached")
        elif should_escalate(state):
            state.stop(reason="human_escalation")

    return build_verified_result(state)

Notice how little of that is the model call. Tool contracts, authorization, state, error taxonomy, verification, telemetry, recovery, and termination are the parts that determine whether the loop is useful and safe. The single line decision = model.choose_next_action(context) is the part most teams spend the most time on, and it is rarely where production agents actually break.

Core components of AI agent architecture

The pieces below make up a working agent architecture. Most teams build them in roughly this order.

Goal and task contract

The goal describes the desired outcome. The task contract makes that outcome executable by defining inputs, constraints, allowed actions, completion evidence, and failure behavior.

A strong contract answers:

• What result should exist when the run is complete?
• Which systems may the agent read or change?
• Which actions are prohibited or require approval?
• What should the agent do when information is missing?
• How will the runtime verify success?
• When should the run stop or escalate?

Writing the contract before the prompt is the cheapest reliability work available. It is also the step most likely to reveal that the task, as stated, has no observable definition of done. When that happens, the right move is usually to narrow the task rather than to make the agent smarter. Our field notes on avoiding reward hacking and building better specs cover what goes wrong when the contract is loose.

Model or controller

The model interprets inputs and makes decisions that are difficult to encode with ordinary rules. Different steps may use different models based on accuracy, latency, cost, modality, or tool-calling performance.

The model should have the smallest decision surface that still solves the task. A model does not need to own authorization, retries, financial limits, or every branch merely because it can generate a plausible decision.

Instructions and policies

Instructions describe the role, task, available procedures, output format, and error handling. Policies define boundaries that should remain enforceable even when the model behaves unexpectedly.

Keep critical controls outside the prompt when possible. Access checks, approval requirements, environment restrictions, and spending limits should be enforced by code or infrastructure. A policy-driven agent treats those rules as runtime configuration rather than as prose the model is asked to remember.

Tools and skills

Tools connect the agent to data and actions. A tool can retrieve information, change an external system, execute code, or delegate work. A skill usually groups several lower-level operations into a coherent capability, and skill design turns out to be measurable: we tested six practices for writing effective agent skills and found the differences showed up in eval scores.

Good tool design includes:

• A specific name and purpose
• A small, typed parameter schema
• Clear descriptions and examples
• Distinct boundaries from similar tools
• Structured success and error responses
• Authentication and authorization behavior
• Timeouts, retries, and idempotency where relevant
• Logs that preserve the request, result, and policy decision

Treat tool definitions as an agent-computer interface. Ambiguous names, overlapping tools, hidden side effects, and unstructured errors force the model to infer behavior that the application could define explicitly. Tool-calling accuracy is one of the few agent metrics you can measure cleanly in isolation, and it is usually the first place to look when a run goes sideways.

Interoperability: MCP, A2A, and the shape of the tool layer

Two protocols now shape how most agents get their tools.

The Model Context Protocol (MCP) standardizes how an agent connects to external tools, data sources, and prompts, so a capability can be implemented once and reused across runtimes. The Agent-to-Agent protocol (A2A) addresses the adjacent problem of agents discovering and delegating to one another across organizational boundaries.

Protocols solve distribution, not design. An MCP server with sixty overlapping tools creates the same tool-selection failures as sixty badly named local functions, and it does so with less visibility into what the model actually saw. Scope the tools you expose per task rather than mounting every available server. We ran the comparison directly in MCP vs. CLI skills for agents, and the answer depended on the task in ways the protocol choice alone did not predict.

Environment

The environment is the system the agent observes or changes. It may be a code repository, browser, CRM, data warehouse, ticketing system, local filesystem, simulation, or collection of APIs.

Reliable agents get ground truth from the environment after important actions. A tool return that says “request accepted” may not prove that the downstream state changed. Verification should query the authoritative system when the distinction matters.

Context and retrieval

Context is the information available to the model for the current decision. Retrieval selects external information to add to that context.

The retrieval layer needs its own quality controls. The agent can fail because it searched the wrong source, issued a poor query, applied an incorrect filter, retrieved stale data, or ignored relevant evidence. Those retrieval failures should be distinguishable from reasoning and tool-use failures, which means retrieval quality needs its own evaluators rather than being folded into a single end-to-end score.

State, checkpoints, and memory

These concepts solve different problems:

Every multi-step agent needs state. Long-term memory is optional. Storing every conversation or tool result as “memory” creates privacy, relevance, poisoning, and maintenance problems. Persist only information with a defined future use, scope, owner, retention policy, and deletion path.

This is one of the least settled parts of the stack. Our survey of the AI memory layer found that the available platforms disagree about what memory even is, and our own teardown of memory and state in LLM applications reached the same conclusion. If you are choosing a memory system today, expect to change it.

Orchestrator, runtime, or harness

The runtime connects model calls, tools, state, policies, and termination. In 2026 this layer is increasingly called the agent harness: the control surface that decides what the model sees, what it can call, how failures are retried, when work is delegated, and when the run ends. Depending on the stack, it may provide:

• Graph or event execution
• Retries and fallbacks
• Durable checkpoints
• Sessions and context compaction
• Human approval
• Tool permissions
• Sandboxing
• Subagents and handoffs
• Deployment and scaling
• Tracing and evaluation hooks

The harness returns the most improvement for the least work, because it is the layer you can change without retraining anything. It is also the layer that expires: harness assumptions that made sense for one model generation frequently stop making sense for the next.

The next chapter compares AI agent frameworks and explains how frameworks, SDKs, runtimes, harnesses, and managed platforms divide these responsibilities. For the orchestration vocabulary in isolation, see what agent orchestration means.

Guardrails, permissions, and human oversight

Guardrails constrain inputs, outputs, actions, and data handling. Permissions determine which resources and operations are available for a specific user, task, and trust level. Human oversight provides approval, review, or takeover at defined boundaries.

Useful controls include read-only tools, allowlists, scoped credentials, approval for high-impact actions, isolated execution, parameter-bound approvals, output validation, and circuit breakers.

The distinction between human-in-the-loop and human-on-the-loop is worth making explicit in your design. In-the-loop means a person must approve before the action executes. On-the-loop means a person monitors and can intervene while the agent proceeds. Choosing the wrong one is a common source of both unnecessary latency and unnoticed damage.

Verifiers and evaluators

A verifier checks whether the task reached an observable success state. An evaluator judges the quality or acceptability of an outcome, trajectory, decision, or component.

Use deterministic verification when an authoritative signal exists. Tests, database state, API responses, schema validation, and policy adherence checks are stronger than a model’s self-report. LLM-as-a-judge evaluators are useful for semantic criteria such as relevance, completeness, or adherence to nuanced instructions, but they need to be calibrated against human labels before you trust their scores.

Observability

Agent observability reconstructs the entire run across model calls, retrieval, tools, state changes, handoffs, approvals, errors, latency, and outcomes. Logs from isolated services are rarely enough to explain why a model selected a tool, what context it saw, or how one failure affected later steps.

The unit of analysis matters. A trace covers one run. A session covers the multi-turn interaction a user actually experiences. Agent outcomes are usually decided at the session level, not the request level, which is why session-level evaluation catches regressions that per-span scoring misses entirely.

Read agent observability: how to trace, debug, and improve AI agents for the instrumentation and debugging workflow, and OpenInference for the OpenTelemetry semantic conventions that make agent traces portable across tools.

From ReAct demos to bounded production agents

The original ReAct work combined model-generated reasoning and actions in an iterative loop, then used observations from an environment to guide subsequent steps. It remains a foundational pattern for tool-using agents. The ReAct paper reported gains over prior baselines on HotpotQA and FEVER for question answering and fact verification, and on ALFWorld and WebShop for interactive decision-making. Our walkthrough of the paper covers the mechanics, and how to ReAct to simple AI agents shows the loop in code.

The lesson from early agent products is more specific than “ReAct failed.” Open-ended agents often asked one prompt and one model to handle planning, tool selection, recovery, state, policy, and termination across a very large action space. Tool interfaces were frequently underspecified. Success signals were weak. Runs were difficult to inspect, and small errors could compound across steps.

Production systems have moved toward bounded, hybrid architectures:

• Code owns deterministic rules and high-impact controls.
• Models make decisions where flexible interpretation adds value.
• The available tool set is scoped to the current task.
• State and checkpoints are explicit.
• Actions return structured observations.
• Success is verified against the environment.
• Every run has stop conditions and budgets.
• Traces and evaluations expose failure modes.

ReAct describes a useful loop. It does not provide the complete production system around that loop.

Common AI agent architecture patterns

Architecture patterns differ mainly in who controls the path, how state moves, and when the system can stop. Our breakdown of three production patterns and how to evaluate each one pairs the common ones with the evals that actually catch their failures.

These patterns can be combined. A support system might use deterministic routing, a bounded tool loop for investigation, an approval checkpoint before a refund, and an evaluator that checks the final customer response.

Routing is one useful pattern, not a universal center of every agent. Choose the least open-ended control model that supports the task. For a longer treatment of how these compose, see our agent architectures breakdown.

Types of AI agents

Two taxonomies are commonly mixed together: classical AI agent types and modern LLM system patterns. Both are legitimate. They answer different questions.

The 5 classical AI agent types

The five-type taxonomy that appears in most search results comes from Russell and Norvig’s Artificial Intelligence: A Modern Approach, which groups agents by how they select actions:

• Simple reflex agents respond to the current observation with condition-action rules.
• Model-based reflex agents maintain an internal representation of the environment.
• Goal-based agents select actions that move toward a specified goal.
• Utility-based agents compare possible outcomes using a utility function.
• Learning agents improve a policy or internal model from experience.

These categories remain conceptually useful, and a modern LLM agent is usually a goal-based agent with model-based state. But the taxonomy predates tool calling, retrieval, and language models by decades, and it does not describe the operational stack you have to build and run. Use it to reason about decision-making. Use the patterns below to reason about architecture.

Modern LLM agent types

Tool-using agents

A model selects from APIs, functions, search, code execution, or other tools. This is the smallest common agent pattern and often the right starting point.

Workflow or graph agents

The application encodes explicit nodes, transitions, and state while reserving selected decisions for models. This pattern fits regulated or business-critical processes that need clearer control.

Planner-executor agents

A planner decomposes the goal and an executor performs the steps. The system may replan after new observations or failures.

Research agents

The agent searches, retrieves, compares, and synthesizes information across several rounds. Strong research agents preserve source provenance and evaluate whether evidence supports each claim.

Coding agents

The agent inspects a repository, edits files, runs commands, and uses tests or linters as environmental feedback. Coding is a strong fit because many intermediate and final outcomes can be checked automatically. Cursor’s approach to verifying AI-written code is a good example of what that verification layer looks like at scale.

Browser or computer-use agents

The agent interacts with a visual interface when a direct API is unavailable. These agents need sandboxing, confirmation for consequential actions, and defenses against malicious instructions embedded in pages or documents.

Durable agents

The system runs across long time windows, waits for events or approvals, survives process restarts, and resumes from checkpoints. See long-running agents for the state and recovery requirements.

Multi-agent systems

Several agents divide work through a manager, handoffs, a graph, or parallel execution. Use multiple agents when specialization, isolation, different permissions, or parallelism produces a measurable improvement. A collection of role prompts alone rarely justifies the coordination overhead, and it introduces cascading failures that are hard to attribute after the fact.

Do AI agents learn?

Most deployed LLM agents do not update model weights while completing a task. They adapt within a run by using context, state, retrieved information, and tool feedback. Teams improve them between versions by changing prompts, tools, policies, retrieval, models, orchestration, or training data.

A system with long-term memory can preserve information across sessions without learning a new policy. Reserve the word “learning” for systems that actually update behavior from experience through training, reinforcement, online optimization, or another defined mechanism. Self-improving agents are the case where the loop is closed deliberately: production evidence feeds an optimization step that changes future behavior, and that step is itself measured.

Examples of AI agents

A useful example includes the task, action surface, verification signal, and human boundary.

The strongest use cases share a practical property: the environment provides feedback. Tests, system state, citations, reconciliations, and policy checks let the agent determine whether progress is real.

For production accounts of how teams actually shipped these, see inside Typeform’s AI agent stack, how Booking.com scales AI observability, and how we rebuilt our own support workflows.

When should you use an AI agent?

Use the lowest-complexity system that can meet the task’s quality and control requirements.

A practical decision ladder

1. Start with one model call. Add clear instructions, structured output, examples, and deterministic validation.
2. Add retrieval. Use RAG when the primary problem is missing or changing knowledge.
3. Build a deterministic workflow. Encode the sequence when the required steps and branches are known.
4. Add bounded model decisions. Let a model classify, extract, route, or choose among a small set of actions.
5. Use a model-directed agent loop. Add it when the number or order of steps cannot be fully predicted and environmental feedback can guide progress.
6. Add multiple agents only after the single-agent design reaches a clear limit. Prove that specialization, isolation, or parallelism improves a measured outcome.

Each rung costs something real. A tool loop that averages eight model calls per task costs roughly eight times a single call and adds eight opportunities for the run to go wrong. That is often worth it. It is worth it less frequently than the current level of enthusiasm suggests.

An agent is a good fit when

• The path to the result varies by request.
• The system must interpret unstructured or ambiguous input.
• The task requires several tools or information sources.
• Later decisions depend on results from earlier actions.
• The environment provides evidence the system can use to make progress.
• Success can be verified.
• The action surface can be bounded with permissions and approvals.
• The expected quality gain justifies additional latency, cost, and operational complexity.
• A safe fallback or human escalation path exists.

Prefer a simpler system when

• The path is stable and can be expressed as ordinary code.
• A single model call or fixed RAG pipeline meets the quality target.
• Rules or calculations require exact deterministic behavior.
• The system cannot reliably tell whether the task succeeded.
• The action is irreversible or high impact and adequate controls do not exist.
• Strict latency or cost targets leave little room for loops and retries.
• Tool access would expose more data or authority than the task warrants.

Five questions to answer before building

1. What observable state proves the task is complete?
2. Which decisions genuinely require a model?
3. What is the smallest set of tools and actions required?
4. Which failures should trigger retry, stop, rollback, or human escalation?
5. How will we trace and evaluate the complete run?

A team that cannot answer those questions is not ready to increase autonomy. If you want the business-side version of this argument to take to a stakeholder, why AI agents need evaluation frames it around failure severity and launch readiness rather than architecture.

Common AI agent failure modes

Agent failures rarely come from one component. A plausible final answer can hide a bad retrieval, an unnecessary action, a policy violation, or a tool call that never changed the environment. Our field analysis of production failures found the same pattern repeatedly: the output looked fine and the trajectory did not.

A failure taxonomy makes remediation more precise. Wrong tool selection calls for different work than malformed arguments. Bad retrieval calls for different work than a planning failure. Scoring the entire run with one pass/fail label loses that information, which is the practical reason debugging agents starts with the trace rather than the score.

How to build reliable AI agents

1. Define a bounded job

Write the task, completion evidence, allowed actions, prohibited actions, limits, and escalation behavior before tuning prompts. Broad goals such as “handle customer support” should become smaller jobs with explicit operating boundaries.

2. Use deterministic code for deterministic requirements

Authorization, approvals, financial thresholds, schema validation, retries, rate limits, and irreversible transactions should not depend on a model following prose correctly. Keep those controls in code or infrastructure.

3. Minimize the solution space

Expose only the tools and context needed for the current task. Use task-specific skills, scoped credentials, and stable branches. A smaller action space improves tool selection, reduces security exposure, and makes evaluation easier.

4. Design tools for the model and the operator

Tool schemas should be easy for a model to use and easy for an engineer to inspect. Return structured errors that distinguish invalid input, denied authorization, transient dependency failure, and terminal business failure.

5. Separate context, state, checkpoints, retrieval, and memory

Document which system owns each capability. Define schemas, retention, access, and update behavior. Avoid using an ever-growing conversation transcript as the only source of truth.

6. Verify from the environment

Each important action should produce an observation. Completion should be checked through tests, source-system state, reconciliation, or another authoritative signal. Model self-evaluation can supplement that evidence, but should not replace it when deterministic verification exists.

7. Add stop conditions and budgets

Set maximum turns, wall-clock time, tokens, tool calls, retries, delegation depth, and cost. Detect repeated actions or a lack of progress. Make the terminal state visible in traces and user-facing results.

8. Apply least privilege and approval based on impact

Separate read and write tools. Scope credentials to the user and task. Require parameter-bound approval for consequential actions. Sandboxed code, browser, shell, and file operations should have explicit network, filesystem, and secret boundaries.

9. Trace the complete trajectory

Capture the task, model calls, context references, retrieval, tool selection, arguments, outputs, state transitions, policy decisions, approvals, errors, retries, latency, cost, and final outcome in one connected trace. Agent telemetry needs standards for the same reason distributed tracing did: without shared semantics, every framework produces spans that cannot be compared.

10. Evaluate outcomes and behavior

Measure whether the task succeeded and whether the path was acceptable. Useful agent evaluation metrics include:

• Task success rate
• Output quality
• Tool-selection accuracy
• Tool-argument correctness
• Retrieval quality
• Policy compliance
• Trajectory quality and unnecessary-step rate
• Recovery rate
• Human intervention rate
• P50 and P95 latency
• Cost per successful task
• Consistency across repeated runs
• Robustness to perturbations
• Predictability and severity of failures

The last four come from recent reliability research rather than standard benchmark practice. In Towards a Science of AI Agent Reliability, Rabanser, Kapoor, Narayanan, and colleagues argue that a single success metric hides whether an agent behaves consistently across runs, withstands perturbations, fails predictably, or keeps error severity bounded. They decompose reliability into those four dimensions with twelve metrics, evaluate frontier models on GAIA and τ-bench, and find that large accuracy gains produced only small reliability gains. That gap is the practical reason benchmark scores keep diverging from production behavior.

11. Turn production failures into regression tests

Use a closed improvement loop:

1. Trace the complete run.
2. Evaluate the outcome and trajectory.
3. Inspect the failing decision, tool, retrieval, state transition, or policy check.
4. Decide whether the agent failed, the evaluator failed, or the task definition was incomplete.
5. Refine the prompt, tool, context, model, policy, orchestration, or rubric.
6. Rerun the representative dataset.
7. Add the production failure to the regression suite.

This workflow converts one-off debugging into cumulative engineering knowledge. Once the loop is stable, most of it can run automatically: we described the version we run in from production traces to better AI agents, and the CI-facing half in evals in CI.

AI agent security and human oversight

Agents create a wider attack surface than answer-only LLM applications because they combine untrusted input with tools, memory, credentials, and external actions. Important risks include direct and indirect prompt injection, tool abuse, privilege escalation, data exfiltration, memory poisoning, goal hijacking, excessive autonomy, cascading multi-agent failures, and runaway cost from unbounded loops.

Two references are worth reading in full: the OWASP AI Agent Security Cheat Sheet for controls and testing, and our mapping of the OWASP Top 10 for agentic applications to what you can actually detect in traces.

Build security into the architecture:

• Treat content from websites, documents, messages, and tool outputs as untrusted data.
• Keep system instructions and policy enforcement separate from retrieved content.
• Give each agent the minimum tools, resources, and credential scope required.
• Split read operations from write operations.
• Require human approval for irreversible, financial, administrative, externally visible, or high-impact actions.
• Bind approvals to the exact action and parameters being authorized.
• Validate tool inputs and outputs.
• Isolate code, shell, browser, and file operations in sandboxes.
• Prevent secrets from entering prompts, memory, logs, or outputs unnecessarily.
• Make write operations idempotent where possible and provide previews or dry runs.
• Set retry, depth, token, time, and spend limits.
• Record policy decisions and actions in an audit trail.
• Test prompt injection, tool misuse, privilege escalation, memory poisoning, credential theft, data exfiltration, approval bypass, and multi-agent trust boundaries before launch and after material changes.

Human oversight should be designed as part of the control flow rather than added as a generic review step. Decide where a person must approve, where a person can monitor and interrupt, and where automatic execution is acceptable. The threshold should reflect action impact, reversibility, confidence, and the quality of verification. Agent supervision is a design decision with a cost curve, not a compliance checkbox.

Security and interoperability are also becoming standards questions. NIST’s Center for AI Standards and Innovation launched the AI Agent Standards Initiative in February 2026, organized around industry-led standards, community-led open protocols, and research into agent identity, authorization, and security evaluation.

How to observe and evaluate AI agents

Agent evaluation should inspect four scopes:

1. Outcome: did the task reach the required result?
2. Path: did the agent follow an acceptable and efficient trajectory?
3. Decision: were individual choices, tool calls, arguments, retrievals, and state updates correct?
4. Reliability: does behavior remain consistent, robust, predictable, and safe across repeated runs and changing conditions?

The agent-native evaluation framework develops each of those scopes in depth, including why agents are better measured against invariants than against one correct trajectory. The trace supplies the evidence. Evaluators turn requirements into repeatable judgments.

A final response alone is insufficient for many agent tasks. A run can return a polished explanation after calling the wrong tool, leaking data, skipping an approval, or failing to change the source system. Conversely, a run can complete successfully after a recoverable error. Outcome and trajectory need separate evaluation, which is what agent-run evaluation means in practice.

Start with deterministic checks wherever possible. Use human review for ambiguous, high-stakes, or changing criteria. Use LLM-based evaluators for semantic judgments that have been calibrated against reviewed examples. Run repeated trials for probabilistic behavior and compare distributions rather than selecting an architecture from one impressive trace. Offline and online evaluation answer different questions, and you need both before a change is safe to ship.

Continue with:

• Agent observability: how to trace, debug, and improve AI agents
• How to evaluate AI agents: a production workflow
• AI agent tracing and evaluation: the complete developer guide

Where Arize fits

Everything above assumes you can see the run. That assumption is the hard part.

Arize AX and Phoenix are built around the loop this chapter describes. Traces capture the full trajectory across model calls, retrieval, tools, state changes, and handoffs, using OpenInference conventions on top of OpenTelemetry so instrumentation is not tied to one framework. Evaluators score outcome, path, and individual decisions at the span and session level. Experiments compare changes against a fixed dataset, and production failures become regression cases for the next release.

Phoenix is open source and runs locally. Arize AX adds the managed platform for teams running agents in production. If you are comparing options, our survey of LLM and agent evaluation platforms covers the field including our competitors.

Frequently asked questions

What is an AI agent in simple terms?

An AI agent is software that can choose actions toward a goal, use tools to perform those actions, observe what happened, and decide what to do next. It operates within instructions, permissions, and stop conditions defined by developers.

What is agentic AI?

Agentic AI is a broad term for AI systems that can make decisions and take multi-step action with some degree of autonomy. It includes both code-directed workflows with bounded model decisions and more open-ended model-directed agents. “AI agent” names a specific system; “agentic AI” names the category.

What are the 5 types of AI agents?

The classical taxonomy from Russell and Norvig’s Artificial Intelligence: A Modern Approach lists simple reflex agents, model-based reflex agents, goal-based agents, utility-based agents, and learning agents. Most LLM agents are goal-based agents with model-based state. The taxonomy is useful for reasoning about decision-making but does not describe the modern operational stack of tools, retrieval, memory, orchestration, and evaluation.

How is an AI agent different from a chatbot?

A chatbot describes an interface for conversation. An agent describes a control and action architecture. A chatbot may only generate responses, or it may act as an agent by selecting tools, changing external systems, and verifying task completion. Many agents have no chat interface.

What is the difference between an AI agent and an AI workflow?

A workflow follows paths defined in code, a graph, or a state machine. An agent gives a model authority to choose some part of the path, such as the next tool, subtask, or stopping decision. Production systems often combine both approaches.

Is RAG an AI agent?

A fixed retrieve-then-generate pipeline is usually an LLM application or workflow. It becomes agentic RAG when the model decides whether to retrieve, chooses sources, creates queries, evaluates the evidence, and iterates based on what it finds.

What is an agent harness?

An agent harness is the runtime layer around the model: the code that assembles context, exposes tools, enforces permissions, handles retries and delegation, and decides when the run ends. It is distinct from an agent framework, which is a library you build with. The agent harness is where most production behavior is actually determined.

What is MCP and do agents need it?

The Model Context Protocol standardizes how agents connect to external tools and data sources, so a capability can be implemented once and reused across runtimes. Agents do not require it. It reduces integration work when you have many tools or want portability across harnesses, and it does nothing to fix badly designed tools.

Do AI agents need memory?

Agents need enough state and context to make the next decision. They do not always need long-term memory. Session state, checkpoints, retrieval, and long-term memory solve different problems and should be designed separately.

Do AI agents learn from experience?

Most deployed LLM agents do not update model weights during normal use. They adapt within a run through context, state, memory, and tool feedback. Persistent improvement usually comes from engineering changes, new training data, fine-tuning, reinforcement learning, or another explicit optimization process.

Can one LLM be an agent?

Yes. A single model with instructions, tools, state, and an execution loop can form an agent. Multiple agents are an orchestration choice, not a requirement.

What is a multi-agent system?

A multi-agent system coordinates several agents through a manager, handoffs, a graph, or peer communication. It is useful when specialization, parallel work, isolation, or separate permissions improve a measured outcome. It also adds handoffs, state transitions, cost, and coordination failures.

What is ReAct in AI agents?

ReAct is a pattern that interleaves reasoning, actions, and observations. A model chooses an action, receives feedback from the environment, and uses that feedback to decide what to do next. Production systems add the runtime controls, permissions, state, verification, observability, and evaluation needed around that loop.

Are AI agents fully autonomous?

Autonomy exists on a spectrum. Some agents choose among a few read-only tools. Others can run for long periods or change external systems. Developers determine the operating envelope through orchestration, tool scope, permissions, approvals, and termination rules.

Do I need an AI agent framework?

No. A focused tool-calling loop can be implemented directly with a model API and application code. Frameworks become useful when the system needs graphs, events, durable checkpoints, human approval, multi-agent coordination, deployment primitives, or common tracing hooks. Choose based on the responsibilities the framework takes over and the behavior you can evaluate.

How much does it cost to run an AI agent?

Cost scales with the number of model calls per task, not the number of tasks. A tool loop averaging eight calls per run costs roughly eight times a single-call application before retries, and oversized context multiplies that further. Measure cost per successful task rather than cost per call, because a cheap run that fails and gets retried is the expensive one.

How do you evaluate an AI agent?

Define observable success, build representative tasks, trace complete runs, and evaluate the outcome, trajectory, individual decisions, policy compliance, recovery, latency, cost, and repeated-run reliability. Add production failures to a regression dataset and compare every material change against it.

What are the main risks of AI agents?

The main risks include incorrect actions, unbounded loops, prompt injection, tool abuse, excessive permissions, data leakage, memory poisoning, missing approvals, unreliable recovery, and failures that propagate through multi-agent systems. Least privilege, sandboxing, validation, verification, human oversight, tracing, and adversarial testing reduce those risks.

Next chapter: AI agent frameworks

The architecture should come before the framework. Once the task, control model, state, tools, permissions, stop conditions, and evaluation plan are clear, compare which parts of the runtime your team wants to build and which parts a framework should provide.