There’s one failure that Stuart Sy, a member of OpenAI’s technical staff, keeps returning to. In one system he observed, roughly one in every 30 to 50 conversations would come to a halt, without an error being thrown or an obviously wrong answer appearing in a dashboard.
Any developer who has shipped an agent to production is familiar with this problem: the bug that only fires often enough to matter and hides the rest of the time. But that frequency is exactly what lets it slip through. Once every few dozen runs may be too infrequent to meaningfully move an aggregate quality score, but it is far too frequent to ignore in production.
Sy’s diagnosis starts outside the model. “For the most part, model intelligence isn’t the bottleneck,” he says. The engineering challenge is understanding what happened around the model, measuring the failure, and using that evidence to improve the system.
Why do capable models still fail?
“For the most part, model intelligence isn’t the bottleneck,” Sy says. The gap is a capability overhang: models are already able to do the work teams want, but they are “usually constrained by just the actual tools or contexts that they have access to.”
When an agent fails, start by asking what the run actually saw and what it was allowed to do before deciding whether to swap models.
The same pattern appears across agent types. A support agent running on a strong model will still fail when it cannot reach the account state or call the tool that would have resolved the ticket. A coding agent can reason its way to the correct change and still fail when the harness never exposes the test runner or grants permission to write the file. In both cases, the failure may have less to do with the model’s underlying capability than with whether the surrounding system gave it what the task required.
Better models cannot compensate for every failure in the surrounding system. Context determines what information the agent receives, while the harness governs the tools, state, and actions available to it. A model upgrade alone will not repair a missing tool, truncated context, or silent permission failure.
Before you change models, ask these four things:
- Did the agent receive the documents, state, and tool results it needed?
- Were the tools it needed available, with schemas the model could use?
- Did a tool call fail, time out, or return an empty payload that the model treated as fact?
- Would a human with the same context and tools have finished the job?
If the failure is in one of those layers, moving to a larger model may raise cost without fixing the underlying problem.
Task-specific models gave way to context, prompts, and evals
Sy’s second point is about where engineering time should go now.
“Before, if you wanted to just classify the sentiment of a piece of user feedback or you wanted to put something inside a structured taxonomy, you would have to train a specific model,” he says. That produced “a pretty efficient, but single use, tool.”
The replacement is a foundation model plus the surrounding system: “the right context engineering, prompting, and correct eval set to hill climb on how you set up the prompts and tools.”
Hold the model constant long enough to learn from it, then vary the inputs and the tools around it and measure whether each change actually moved the result.
Context engineering is the practice of deciding what the model sees: facts, session state, tools, and the format that wraps them. Prompt wording still matters, but it is only one layer. The rest is which evidence you retrieve, how you compress it, which tool schemas you expose, and whether you can replay the same bundle after a failure.
An eval set gives that iteration loop a stable target. It defines what success looks like before you start changing prompts, context, or tools. A team replacing a sentiment classifier should evaluate against a labeled sample of real feedback, not a generic helpfulness score. A team replacing a taxonomy model should check whether the new setup routes cases to the right nodes and abstains when the label is unclear. The eval tells the team whether the replacement actually preserved or improved the behavior they care about.
A tight loop looks like this:
- Collect a small set of real examples, including the intermittent failures.
- Write a narrow criterion that names the failure.
- Change one thing in context, tools, or the prompt.
- Rerun the same cases.
- Keep the change only if the eval moves.
Recursive self-improvement is a feedback loop you have to build
“A big theme, both in research and applied organizations, is the concept of recursive self-improvement,” Sy says. You need to ask, “How do you set up these automatic loops that you can hill climb to get better and better?”
Here, self-improvement does not require an agent to autonomously rewrite itself or update its own weights. The loop can be an engineered process that turns production evidence into changes to context, prompts, tools, evals, or the surrounding system.
OpenAI’s version of that loop starts with volume. “As we approach a billion users on ChatGPT, we constantly get a huge stream of feedback from so many different sources,” Sy says. “It’s more than anyone can manually read through, understand, prioritize, and take action on all at once.”
So they built a system to process millions of data points per day, then create a shared understanding of that stream. “You definitely need some heavy duty traditional software engineering here just to handle all of that volume,” Sy says.
Most teams will not process millions of events a day. They can still apply this process:
- Put ratings, tickets, chat corrections, and trace failures into one event model, with the raw text and a conversation or trace id attached.
- Bootstrap a short taxonomy from the failures you already know: tool error, missing context, wrong label, abandoned session.
- Cluster recent leftovers so new failure modes can appear before someone invents a chart for them.
- Promote a confirmed cluster into an eval case, then watch it after you ship.
Why agent evals need process metrics
As agent runs get longer, final-output scores hide more of the failure.
“You want to pay more attention to not just the final output, but what’s happening in the process,” Sy says. Developers should be able to answer:
- What tool calls were made, and how many?
- Is there an error rate and latency associated with those calls?
- Does the amount of thinking effort the model uses affect the final output?
A coding agent might produce a correct-looking patch after twelve failed tool calls, three timeouts, and repeated attempts that never incorporate the test results. An eval that looks only at the final diff can miss those process failures. A trace exposes them.
This is also why harness evaluation sits next to model evaluation. The harness is the runtime that manages tools, context, state, retries, and stopping conditions. If the trace shows a high tool-error rate, the next change may be the tool schema, the retry policy, or the permission layer, not the system prompt.
Hamel Husain makes a related point in the second installment of this series: generic metrics rarely capture the failure that actually matters, and useful evals start by looking at real data. Sy adds the instrumentation requirement. You cannot look at the data if the run never recorded the tool calls, errors, latency, or thinking effort.
Practically, that means instrumenting the trajectory rather than reviewing only the final response. In Arize Phoenix or Arize AX, for example, you can inspect:
- Spans for each model call and tool call, including arguments, errors, and latency.
- Trace-level views for one request, so you can see the path rather than only the last message.
- Session-level evals when the work spans multiple turns.
- Production traces for the rare case, such as the chat that stops one time in 30, which a 20-example golden set will never catch.
That 1-in-30 stop is exactly the kind of failure production tracing can uncover. Offline evals remain necessary, but production runs expose rare and emerging failures that a fixed eval set may not contain yet.
An AI engineer still has to be an engineer
Near the end of the conversation, Sy finishes the same sentence this series keeps asking: an AI engineer is someone “who builds AI products or systems, using the new set of primitives now available to us on top of these foundation or large language models.”
Then he puts a constraint on it. “A strong foundational engineering skill set is still required. You still really want to understand the existing systems you’re building on top of.”
AI engineering builds on those existing engineering skills with a new set of concerns around models, context, tools, evaluation, and agent runtimes. An AI engineer should know how to “squeeze out the most performance from Gen AI” and use it “in a principled and predictable way.” In Sy’s list, that means context engineering, harness engineering, the right tools, skills, and MCP connectors, and the ability to observe agents well enough to deliver production-level performance.
That is a sharper job description than “prompt engineer.” It also matches what Michael Grinich argued in the first installment: the systems around the agent now decide whether autonomy is useful or merely fast at failing.
“Principled and predictable” is a useful bar for the discipline Sy describes. Replaying failed runs makes them reproducible. Tool errors and latency help separate context failures from runtime failures. Turning recurring production complaints into eval cases gives teams a way to test whether the next change actually fixes the problem.
You can watch the full Rise of the AI Engineer episode with Stuart Sy.
