The gap between an issue showing up in production and a fix going out is the constraint most teams shipping agents are now working against.
Most of those teams have stopped asking whether they need observability and evals, they already have both. Traces are landing, evaluators are running, and the dashboards are populated. That is exactly why the gap is wide: an application with thousands of users a day produces more traces and eval results than any team can read, and every hour spent combing through them is an hour the fix isn’t shipping.
Jim Bennett, Principal Developer Experience Engineer, sat down with SallyAnn DeLucia, Head of Product, to talk about what happens when AI telemetry outgrows the humans reviewing it. Her answer is not fewer people. It is humans managing a fleet of specialized agents that cut the pile down, so the people with domain knowledge only spend time where they change the outcome.
“Teams are not lacking observability, not even lacking evals anymore. Everybody pretty much seems to have that,” SallyAnn said. “But the big question I’m getting is what do we do with this? And how do we make it the most efficient?”
How did the AI debugging loop work before agents?
Until a few months ago the loop looked the same everywhere. A monitor fires or a metric drops. A human investigates, works out the pattern and the root cause, takes it into an agentic IDE, makes a change, pushes it, and tries to test it in the loop.
Every step sat behind one person’s attention, and the worst of it was combing traces to find the pattern, which can take hours and scales worse as traffic grows.
What is changing is who does the first pass. “Instead of the human being the one that goes in and does that initial investigation, or even being alerted, that’s where the agent’s starting to come in,” SallyAnn explained. “Something that would take a human a lot of time, maybe even hours, an agent can really do in a few seconds, a few minutes.”
Signal in Arize AX is an agent built to do exactly that first pass. It scans a tracing project on a schedule, groups recurring failure patterns into issues, and writes an investigation for each one: an overview, the trace IDs to look at, and a proposed fix to a prompt, code, configuration, or evaluator. With a repository attached it carries the investigation into the codebase and opens a pull request.
The agent puts up a fix, then a human reviews it.
What high-leverage work actually means
The interesting part of that shift is not what the agents take on, it is what the humans get back.
“Combing through data is very tedious. I wouldn’t really actually call that high leverage,” SallyAnn said.
High-leverage work is work where one human decision changes the output of many agent runs. Annotating a span teaches every eval derived from it. Combing through traces teaches nothing, it just moves the pile from one place to another.
Four things qualify: annotating data so your feedback is grounded in human judgment, curating datasets to run experiments against, reviewing the agents’ work, and nudging agents in different directions.
Annotation is the hardest of the four, because it doesn’t scale by adding reviewers.
You can’t ship thousands of production traces to a person and ask them to check all of them. What an agent can do is review them and come back with a short list: “these are the anomalies I think I’m seeing, based on my understanding of the system. Please annotate these.”
The domain expertise is still firmly in the realms of humans. The agent is only reducing the size of the pile. As SallyAnn put it, it shifts the job “from being where’s the needle in the haystack, to here is a set of things that actually matter, that are actually worth your attention.”
This matters most when your annotators are not on your engineering team. For example, education software companies advertise for teachers to annotate their AI data, because the teacher in the classroom knows whether an output is any good in a way an engineer doesn’t. Employing thousands of teachers to scan raw traces isn’t a plan. Giving a handful of them a small set of flagged traces to work through part-time is.
Eval alignment is the clearest example
Aligning an evaluator to human judgment has a lot of steps and one irreplaceable human input.
Today a person has to notice the eval needs aligning, curate the dataset, apply the ground truth, adjust the judge prompt, iterate several times, and push it. Most of that is mechanical, but the ground truth is not.
“I like the kind of interaction between the agent and the humans where the agent can say I did my part, now I need you to go do your part,” SallyAnn noted. The agent watches the evaluator, decides it has drifted, and curates exactly the spans that need a label. The human applies the annotations. Alignment happens.
Or, as Jim put it in the conversation, the agent is the over-enthusiastic intern. It says “I think this, I think this, I think this.” It does the basics. The human reviews the work, because the human has the knowledge.
What types of managed agents make up an AI operations fleet
Signal is an agent with one role. SallyAnn walked through the others Arize is building as managed agents:
- Eval alignment: curating the spans a subject matter expert needs to label.
- Eval quality: watching for evaluators that are unhealthy: an LLM as a judge costing real money for a job that should be code, or an eval that always returns the same label and is therefore giving you no signal. It also flags the evals you are missing.
- Cost: watching spend across models and evaluators and flagging where you are overpaying, on the theory that you spend a little to save a lot.
- Dataset coverage: keeping datasets current and checking they still cover the use cases you care about.
- Safety and security: watching across every project rather than a single use case. The viral post about an OpenAI agent getting into Hugging Face made this a live question for a lot of teams.
- Classification: categorizing incoming data along several axes so the other agents have something structured to work from.
All these agents have the same dependency. They need high-quality data to run on. “Our traces, our logs, our evals, they’re not going anywhere,” SallyAnn said. “It’s just who actually consumes that is what’s changing.”
The trajectory is fast. Most teams run one agent today, usually Signal. SallyAnn expects two or three within a month, and potentially dozens on top of a single application within a few months.
The human doesn’t go away
The first question everyone asks is “will the human go away?” The answer is unequivocally no.
The volume of work the human has to deal with is going up, because we are capturing more traces, capturing more evals, and building more agents. Dropping agents into the parts of that work that don’t need a brain is what frees the human up for the parts that do.
The job description is mostly what changes. It stops being “look at the telemetry” and becomes “what are my agents doing, are they doing the right things, and is there anything they need me for?”
What skills should you look for in an AI Operations role?
If you are an engineering manager building this team, the fundamentals get more important. Systems design and first principles thinking are how you keep a fleet of agents on track. Subject matter expertise is the only thing grounding those agents in what humans actually expect. You probably have both on the team already.
One skill is new: managing a team of agents. We wouldn’t hire someone today because they are good at digging through millions of rows of data. We would hire someone who can send an agent to do it, or a team of agents.
The reason to care is iteration speed. Coding harnesses made building fast. Fixing didn’t keep up, and closing the gap between issue and fix is the constraint teams are now optimizing against. We see it internally: the team building Alyx leans on Signal to surface things they weren’t focused on and to open PRs that would otherwise have taken a slice of their week, which they spend on bigger bets instead.
The advice is unglamorous. Turn the agents on, run them against a real project, and get used to reviewing their work. Start with Signal in Arize AX and add roles as they land.
