Harness engineering: how to build reliable AI agents

Key takeaways

• Start with a task contract that makes the outcome, scope, invariants, budgets, and escalation conditions explicit.
• Design completion gates before execution so the harness can prove success from current system state rather than a model summary.
• Keep authority deterministic: the model may discover a path, but code and policy own permissions, side effects, and finish conditions.
• Anchor checkpoints to the repository or external state that produced them, and reject stale or incompatible recovery state.
• Choose retry and recovery behavior by side-effect class; ambiguous writes and irreversible actions require reconciliation, not blind repetition.

This guide starts where harness anatomy ends. Once an agent runtime can call tools, maintain state, and loop, the reliability problem becomes concrete: what job is allowed, what side effects are permitted, what evidence proves completion, and what should happen after an interrupted or ambiguous action.

Those controls should be enforced by runtime code and policy rather than left to the model’s memory or prose instructions. The sections below focus on execution reliability. For the broader architecture, tracing, evaluation, and improvement loop, use Arize’s agent harness architecture, tracing, and evaluation guide.

What does harness engineering mean in this guide?

Harness engineering is the task-specific work of governing an agent runtime so a production job is bounded, verifiable, resumable, and recoverable without prescribing the path the model must take. The harness supplies the loop and operational primitives; harness engineering defines the contract, authority, evidence, state, and recovery rules for the job.

An agent harness is the runtime; harness engineering is the reliability work that makes that runtime dependable for a specific class of work. The distinction matters because these controls define the execution envelope, not the trajectory through it. The model can still search, choose tools, compact context, delegate work, and adapt its next step to intermediate evidence. Deterministic code and policy retain control over what may execute, what side effects are permitted, what state can be trusted, and what counts as finished.

For the nine-component anatomy and the framework-versus-harness distinction, see What is an agent harness?.

The term also sits inside a broader shift toward agent-driven software engineering. OpenAI uses “harness engineering” for designing environments, specifying intent, and building feedback loops around coding agents. Andrej Karpathy’s autoresearch makes a bounded loop concrete with one agent-editable training file, a fixed five-minute run budget, a single objective metric, and a keep-or-discard cycle. Hermes, Nous Research’s open-source agent harness, shows a complementary runtime pattern with durable sessions, controlled tool exposure, context compression, and long-running execution.

Regardless of terminology, the engineering boundary is the same: prompts can influence the model, but contracts, permission checks, state verification, and completion gates must be enforced outside the model when they need to be guaranteed.

Harness engineering does not replace agent autonomy with a predefined workflow. It specifies the conditions that must remain true while the model determines how to get there. A workflow can prescribe the path; an agent harness can let the model discover the path while retaining deterministic control over authority, state, side effects, and completion.

What makes a harness reliable in production?

For this guide, reliability means six properties. They are intentionally execution-focused so the page does not duplicate the separate Arize guide to harness tracing and evaluation.

The reliable harness engineering control loop

A reliable harness lets the model choose the next useful action while the runtime owns the contract, authorization, completion evidence, checkpoint state, and recovery decision.

The loop begins with a task contract and ends only when completion gates pass against current state. Between those points, the model can adapt the path. After each material side effect, the runtime records enough verified state to resume or reconcile without turning “try again” into the default recovery strategy.

How do you engineer a reliable harness?

The following sequence works across coding agents, support agents, data-analysis agents, research agents, and internal automation. The examples use a coding agent asked to add per-client API rate limiting because the artifacts and side effects are easy to inspect.

1. Define a task contract before the model starts

Translate the user request into a compact contract that the runtime can inspect. The contract should specify the outcome, allowed scope, invariants, completion evidence, budgets, and escalation conditions. Keep it short enough for a developer to review and structured enough for code to enforce.

What follows is an example task contract. The names should map to real validators in the harness.

task:

id: add-api-rate-limiting

outcome: “Return HTTP 429 after a client exceeds the agreed limit”
allowed_changes:

– middleware/**

– config/rate_limit.*

– tests/rate_limit/**
forbidden_changes:

– auth/**

– public_api/**

– database_schema/**
invariants:

– existing_authorization_tests_pass

– unrelated_required_tests_pass
completion_gates:

– targeted_tests.exit_code == 0

– required_suite.exit_code == 0

– diff.outside_allowed_scope == false

– response_contract.status == 429
escalate_when:

– new_dependency_required

– schema_change_required

– access_outside_sandbox_required

Completion principle: A convincing final message is not evidence. The harness should read the current test report, inspect the current diff or external record, verify required approvals, and reject completion when evidence is missing, stale, or outside the task scope.

2. Design completion gates around the side effect

3. Separate model discovery from deterministic authority

Use a model when the next useful action depends on ambiguous evidence. Use deterministic code or policy when an unacceptable outcome can be stated exactly. This division lets the agent adapt without allowing natural-language instructions to become the authorization system.

This is the same boundary established earlier: the model owns discovery inside the task envelope; deterministic controls own authority and verification.

Engineer guarantees, not compensations for model weakness

The best harness controls remain useful as models improve. Authorization boundaries, completion evidence, idempotency, durable state, and reconciliation protect system invariants regardless of model capability. By contrast, elaborate routing rules or prompt choreography added only to compensate for a model’s current limitations may become unnecessary as models improve.

Treat those two classes differently. Make guarantees stable and explicit. Keep model-specific scaffolding easy to remove, replace, and evaluate.

4. Bound capabilities and permissions

A capability interface does not replace the harness control plane. MCP standardizes how hosts and clients connect to servers that expose tools, resources, and prompts. Skills package reusable task procedures. CLIs and APIs expose execution surfaces. The harness still decides which capability is visible for the current task and whether a proposed invocation is valid and authorized.

MCP includes protocol-level authorization for supported HTTP deployments, but authorization is optional in the protocol and does not encode your organization’s business policy. Tenant boundaries, data classification, action risk, and user-specific authority still belong in the host, server, or a separate policy layer.

What Arize’s 500-run benchmark showed

On the hardest GitHub analysis tasks, a thin MCP surface averaged about 12 tool calls versus 5 for skills, with more than 6x the cost and 5x the latency. MCP tool fidelity was 0.33. The result reversed when the task mapped cleanly to endpoints: creating a branch and pull request averaged 8 calls with MCP versus 22 with a verbose skill. The lesson is to select capabilities by task shape, composition, authorization, and deployment context rather than declaring one interface universally superior.

Source: MCP vs. CLI Skills for agents: what our eval found. The experiment used GitHub tasks and a thin REST-style MCP server, so the measured ratios should not be generalized to every MCP architecture.

Use both interfaces and let the harness choose by task shape. Use the CLI for local workflows, tools with deep training-data coverage, and work that benefits from composition. You should use MCP when the tool is remote or proprietary, when you need OAuth and per-user authorization, when real state spans steps, or when you want an entire agent behind a single tool call. The harness, not the prompt, should decide which surface is visible and authorized for the current stage.

Enforce policy outside the model

A pre-execution policy decision should validate more than a tool name. Check the requesting principal, user or tenant context, capability, resource, arguments, data classification, environment, and current state. A post-execution hook should capture the actual side effect, redact sensitive output, update durable state, and trigger the relevant evaluator or monitor.

External enforcement limits what a prompt injection can cause; it does not eliminate prompt injection. An injected instruction can still persuade the model to request an action that falls inside an overly broad permission rule. Use least privilege, tool-specific parameter validation, sandboxing, output handling, monitoring, and human approval for high-risk or irreversible operations.

5. Make every long-running unit resumable

A long-running task can outlive a model call, worker, browser session, sandbox, or deployment. Divide it into units that end with a durable record of the input version, action, output artifact, verification result, external side effects, and next eligible step. Resume from the last verified unit rather than from the model’s memory of the conversation.

For the rate-limiting change, useful units are: reproduce the current behavior, add a failing test, implement the change, run targeted tests, run the required suite, inspect scope, and prepare the delivery artifact. Each unit should be independently verifiable and safe to skip when its evidence still matches current state.

Checkpoint the workspace instead of just the commit

A Git commit ID does not represent staged changes, unstaged changes, or untracked files. The example below requires a Git repository with at least one commit, resolves the repository root before every Git read, rejects a checkpoint directory inside the worktree, fingerprints HEAD plus index, worktree, and untracked state, hashes referenced artifacts, and rejects unknown checkpoint schemas.

As an example, in Python 3.9+ you should verify repository state and artifacts before resuming, independent of the caller’s current directory.

A useful checkpoint should bind recovery state to the actual workspace that produced it, not merely to a conversation or commit ID. For a coding agent, that can include HEAD, index state, worktree changes, untracked files, artifact hashes, and a checkpoint schema version. On resume, reject the checkpoint if any of those inputs no longer match.

“””Repository-scoped checkpoint for a coding-agent harness. Python 3.9+.”””

from __future__ import annotations
import hashlib

import json

import os

import subprocess

from pathlib import Path

from typing import Dict, List, Optional
SCHEMA_VERSION = 1
# Keep agent state outside the worktree so the checkpoint cannot

# invalidate its own workspace fingerprint.

AGENT_STATE_DIR = Path(os.environ.get(“AGENT_STATE_DIR”, “/tmp/agent-state”))

CHECKPOINT = AGENT_STATE_DIR / “checkpoint.json”
def repo_root(start: Optional[Path] = None) -> Path:

“””Resolve the git root so fingerprints stay stable across cwd changes.”””

start = start or Path.cwd()

out = subprocess.check_output(

[“git”, “-C”, str(start), “rev-parse”, “–show-toplevel”],

text=True,

).strip()

return Path(out)
def _git(root: Path, *args: str) -> str:

return subprocess.check_output([“git”, “-C”, str(root), *args], text=True)
def workspace_fingerprint(root: Optional[Path] = None) -> str:

“””Fingerprint HEAD + index tree + worktree/untracked status.
Requires a repository with at least one commit (`git rev-parse HEAD`

exits non-zero in an empty repo).

“””

root = root or repo_root()

head = _git(root, “rev-parse”, “HEAD”).strip()

index = _git(root, “write-tree”).strip()

status = _git(root, “status”, “–porcelain=v1”, “-uall”)

payload = f”head={head}\nindex={index}\nstatus=\n{status}”

return hashlib.sha256(payload.encode()).hexdigest()
def artifact_hashes(paths: List[str], root: Optional[Path] = None) -> Dict[str, str]:

root = root or repo_root()

return {

rel: hashlib.sha256((root / rel).read_bytes()).hexdigest()

for rel in paths

}
def save_checkpoint(state: dict, root: Optional[Path] = None) -> Path:

“””Atomic write: temp file beside the checkpoint, fsync, then os.replace.”””

root = root or repo_root()

state_dir = AGENT_STATE_DIR.resolve()

if str(state_dir).startswith(str(root.resolve()) + os.sep):

raise ValueError(“AGENT_STATE_DIR must not live inside the worktree”)
state_dir.mkdir(parents=True, exist_ok=True)

tmp = CHECKPOINT.with_suffix(“.json.tmp”)

payload = json.dumps(state, indent=2, sort_keys=True).encode()

with open(tmp, “wb”) as f:

f.write(payload)

f.flush()

os.fsync(f.fileno())

os.replace(tmp, CHECKPOINT)

return CHECKPOINT
def load_checkpoint(root: Optional[Path] = None) -> Optional[dict]:

“””Return state only when schema, workspace, and artifacts still match.”””

root = root or repo_root()

if not CHECKPOINT.exists():

return None
state = json.loads(CHECKPOINT.read_text(encoding=”utf-8″))

if state.get(“schema_version”) != SCHEMA_VERSION:

return None # unknown or mismatched schema — non-resumable
if state.get(“workspace”) != workspace_fingerprint(root):

return None # HEAD, index, worktree, or untracked files changed
for rel, digest in state.get(“artifacts”, {}).items():

path = root / rel

if not path.exists():

return None

if hashlib.sha256(path.read_bytes()).hexdigest() != digest:

return None
return state
# Example: persist verified progress after targeted tests pass.

root = repo_root()

save_checkpoint(

{

“schema_version”: SCHEMA_VERSION,

“workspace”: workspace_fingerprint(root),

“completed_units”: [“middleware updated”, “unit tests passed”],

“next_step”: “run integration tests”,

“artifacts”: artifact_hashes([“test-results/unit.xml”], root),

},

root,

)
# On resume: continue only when the workspace still matches.

state = load_checkpoint(root)

if state is None:

raise SystemExit(“stale or incompatible checkpoint — enter recovery”)

The important property is not the storage format. It is that the runtime proves the checkpoint still corresponds to current execution state before trusting it.

Durability and scope

The sample is intentionally local and repository-scoped. It resolves the Git root so the fingerprint does not change when the caller moves into a subdirectory; it requires at least one commit and rejects AGENT_STATE_DIR inside the worktree so the checkpoint cannot invalidate its own fingerprint. A schema_version mismatch is treated as non-resumable state. The temporary file is created beside the checkpoint, flushed with fsync, and then replaced with os.replace. Full power-loss durability can require additional filesystem-specific steps; distributed or business-critical runs should use a transactional state store or durable object store.

6. Attach retry and recovery policy to each stage

A generic retry loop treats every failure as if the operation were safe to repeat. Recovery should depend on side effects and on what the runtime can prove about the previous attempt. Every stage needs a timeout, attempt limit, backoff policy, last verified checkpoint, reconciliation action, and escalation condition.

Polling deserves the same discipline. Prefer an event, webhook, or durable queue. When polling is unavoidable, persist the pending operation, use bounded backoff with jitter, and schedule the next eligible check without keeping the model loop active.

7. Route models and deterministic stages deliberately

The harness should reserve model calls for decisions that benefit from semantic judgment. Retrieval, normalization, exact calculations, schema validation, and invariant checks generally belong in code. Narrow, familiar classifications can use a smaller model; high-impact decisions with conflicting evidence may require a stronger model or human review.

Keep the contract, policy, state model, trace schema, and completion gates stable when the model changes. A model upgrade should be one versioned component of the harness configuration, not a reason to lose comparability across runs.

When should you use a pipeline, bounded workflow, or agent harness?

Use an agent loop only where the path itself must adapt to evidence at runtime. If the path and branches can be specified reliably in advance, prefer a deterministic pipeline or bounded workflow. Harness engineering begins once the model is allowed to choose among possible next actions and the runtime must govern that autonomy.

The presence of a model does not make a system an agent.

A practical default

Most production systems are hybrids. Keep operations whose path can be stated and tested in deterministic code. Use model judgment where the evidence is ambiguous, and use an agent loop where the next useful action cannot be enumerated reliably in advance. Harness engineering governs that adaptive portion without turning it back into a predefined workflow.

How do you verify a harness change before shipping?

Compare the old and revised harness on the same representative tasks and the same task contracts. Keep hard gates for scope, permissions, and completion fixed; then compare task success plus latency and cost on successful runs. When a production failure motivates the change, keep that exact case in the regression set.

This page stops at that promotion gate. The canonical agent harness architecture, tracing, and evaluation guide covers instrumentation, evaluation levels, metric design, failure analysis, and the full trace-to-improvement loop.

Production harness engineering checklist

Harness engineering with Arize

Once the runtime owns contracts, gates, checkpoints, and recovery, telemetry provides the evidence needed to prove those controls actually fired. Instrument tool actions, policy decisions, state transitions, retries, approvals, and final artifacts so false completion, duplicate side effects, stale resumes, and policy failures are diagnosable.

Arize Phoenix provides an open-source workflow for tracing, evaluations, datasets, prompt iteration, and experiments. Arize AX adds managed production workflows for observing agents, running online and offline evals, curating datasets, comparing experiments, monitoring quality, and investigating recurring failure modes. Those capabilities let teams test reliability changes against the same traces and regression cases that exposed the failure.

Use this page to design the execution controls; use the agent harness architecture, tracing, and evaluation guide for the end-to-end observability and evaluation workflow. A practical starting point is to send a representative trace, verify one completion or safety gate from observed behavior, and save the first real failure as a regression case.

Frequently asked questions about harness engineering

What is the difference between an agent harness and harness engineering?

An agent harness is the runtime architecture that manages the model loop, context, capabilities, state, permissions, and execution. Harness engineering is the task-specific work of governing that runtime: contracts, policy, completion gates, durable state, recovery, and routing. You can use a packaged harness and still perform substantial harness engineering.

Does harness engineering replace prompt engineering?

No. Prompts remain useful for task instructions, examples, rubrics, tool descriptions, and model judgment. Harness engineering adds controls that a prompt cannot guarantee, including authorization, schema validation, budgets, idempotency, durable state, completion evidence, and external verification.

Does MCP make agent tool use secure?

MCP standardizes connections and defines security requirements for tools and resources. Supported HTTP deployments can use MCP authorization, but authorization is optional in the protocol and transport-level access is only one layer. The application still needs least privilege, tenant-aware policy, parameter validation, output handling, approval for high-risk actions, monitoring, and audit records.

How do you make a long-running AI agent reliable?

Break the task into resumable units. At each unit, persist the input version, action, verified output, side effects, artifacts, and next eligible step. On restart, compare the checkpoint with current workspace and external state before continuing. Use operation IDs or idempotency keys for writes, and reconcile ambiguous outcomes before retrying.

How do you know when an AI agent is finished?

Define completion gates before execution. Gates should inspect current evidence such as test reports, scoped diffs, external records, operation IDs, approvals, and artifact hashes. The harness reaches the finish state only when all required outcome and safety gates pass against the current version of the work.

How do you test a harness reliability change?

Run the old and revised harness against the same representative tasks and task contracts. Hold hard scope, permission, and completion gates constant; compare successful-task rate, latency, and cost; and preserve real failures as regression cases. For evaluator design and trace-level analysis, use the harness tracing and evaluation guide.

Related Arize resources

• What is an agent harness? The canonical Arize explanation of harness anatomy and the nine-component model.
• Context management in agent harnesses: A focused guide to memory, files, compaction, tool outputs, and subagent context.
• MCP vs. CLI Skills for agents: The full 500-run benchmark behind the capability-selection findings cited above.
• Agent harness architecture, tracing, and evaluation: The canonical resource for generic agent-harness architecture, tracing, evaluation levels, metrics, and the improvement loop.
• How Hermes implements an open source agent harness architecture: A concrete reference for durable sessions, tool exposure, context compression, and long-running execution.
• Agent evaluation metrics: Metrics for task success, quality, safety, cost, latency, and operating views.