TL;DR: Predefined evals and monitors measure risks you already know how to specify. Production agents also develop failures nobody thought to encode: a tool receives the wrong argument, a retry loop adds cost without making progress, or the final answer looks right even though the requested action never happened. Signal reviews production traces and groups recurring patterns into investigations with supporting evidence. Engineers then preserve representative examples in a regression dataset, define the expected behavior, and create evaluators to test candidate fixes. Those datasets grow as the team confirms new failures, while Signal keeps watching for patterns the existing suite does not cover.
Every eval suite begins with a list of things that might go wrong. But as is often the case, production eventually produces something that wasn’t on your list.
We saw the gap in one of our own controlled benchmarks. We ran seven models through the same agent harness on 19 GitHub operations tasks, ten times each. Correctness stayed in a relatively tight band, from 79.6% to 85.1%, while latency, tool-call counts, retry behavior, and timeout risk moved much more. The full experiment covers the setup, but the lesson for production reliability is that final-answer scores can make systems look similar while their trajectories carry very different cost and failure exposure.
That’s why the path has to become evaluation evidence, and why the eval stack needs a way to discover failures before you even realize they exist.
Known agent failures are only half the problem
An agent trajectory is the observable sequence of work behind an outcome: the original request, available context, model calls, retrieval, tool calls and responses, errors, retries, permission decisions, state changes, artifacts, explicit planning steps when available, and the final answer or action. Output-only evals score the last item and production reliability depends on the full sequence.
Consider an agent asked to investigate an account issue, update a record, and write a summary. The summary can be accurate while the run failed several steps earlier. The agent may have queried stale data, passed an invalid argument, retried the same failed operation, or claimed completion without verifying the updated record.
An output judge can pass the summary while the user didn’t actually get the requested outcome.
Known failures are much easier to encode once the team understands the behavior and can define an observable success or failure condition. A code evaluator can require a successful state change. An LLM judge can score whether a response is grounded in retrieved evidence and a monitor can issue an alert when a known failure rate rises. But it gets more challenging when you think about identifying a recurring path failure you don’t even know exists, or how to measure.
Manual trace review works when the sample is small. But it quickly breaks down when an agent produces thousands or millions of trajectories and the important pattern is spread across runs rather than isolated in one obvious error.
Signal turns trace volume into a prioritized issue queue
Signal is an always-on AI worker in Arize AX. It reviews production traces on a recurring schedule, keeps track of previously identified issues, and surfaces new patterns as they emerge. Instead of returning another stream of traces, Signal groups related runs into investigation reports with a summary, likely root cause, impact, supporting evidence, and a recommended next step with a pull request on GitHub.
That changes the starting point for production agent debugging. Engineers no longer have to begin with a blank trace and search for a pattern manually. They begin with a ranked issue and the traces that support it.
- Reviews behavior across runs instead of treating every trace as an isolated incident.
- Distinguishes known issues from emerging patterns so the same failure is not rediscovered from scratch.
- Keeps the investigation grounded in trace evidence, including the relevant tool calls, retrieval, errors, retries, and outcomes.
- Returns an actionable issue rather than a generic anomaly or threshold breach.
- Provides an on-ramp to deeper workflows with repository context, custom skills, managed agents, and experiments.
Signal is especially useful for failures that look healthy to conventional monitoring. The request may return a 200. Latency may stay inside the service-level objective. The response may even receive a good output score. The trajectory still shows wrong retrieval, bad tool arguments, skipped steps, redundant work, or false completion.
Signal tells you what your eval stack is missing
A production evaluation system needs distinct layers because each layer answers a different question:
| Layer | Best for | Role in the improvement loop |
|---|---|---|
| Code evaluators | Exact, known conditions | Run cheap, repeatable checks for schemas, required steps, state changes, status codes, and policy invariants. |
| LLM-as-a-Judge | Known semantic criteria | Apply a stable rubric to grounding, relevance, instruction adherence, or other meaning-based criteria. |
| Agent-as-a-Judge | Complex multi-step criteria | Explore trace context dynamically, reason across spans, and evaluate complex trajectory-level behavior. |
| Signal | Recurring and emerging production patterns | Continuously review production traces, group related failures, and produce evidence-backed investigations that identify what deserves a new test or fix. |
Signal and Agent-as-a-Judge in Arize AX are complementary. Signal asks which patterns matter across the production population while Agent-as-a-Judge applies a complex judgment to trace context and generate reusable evaluation labels. Once a failure becomes well understood, a narrow LLM judge or deterministic check is usually the better long-term guardrail.
The useful operating principle is simple: keep Signal watching for new failures as they appear while using named tests for failures you’ve already identified.
A recurring Signal issue should graduate into regression protection
Discovery creates value when teams carry production findings into the tests they run before the next release. A practical Signal workflow looks like this:
- Detect. Signal finds a recurring or emerging pattern across production trajectories and opens an issue with supporting traces.
- Confirm. An engineer reviews the evidence, decides whether the behavior is actually wrong, and defines the expected behavior.
- Make it measurable. Preserve the evidence by adding representative examples from the traces behind the issue to a dataset. In Arize AX, use the Add to Dataset workflow on the relevant spans. Record the expected behavior separately from the captured output, include the context needed to reproduce the failure, and pair the cases with a code evaluator, LLM judge, or Agent-as-a-Judge evaluation.
- Investigate and propose. A human engineer or repo-aware Managed Agent examines the harness, prompt, tool, retrieval, routing, or state logic and prepares a candidate change.
- Verify. Run a full-agent experiment against the same fixed set of examples for both baseline and candidate. Compare outputs, tool use, retrieval, latency, cost, trajectories, and eval results.
- Review and ship. Engineers approve, modify, or reject the change through the normal release process, then watch for recurrence in production.
These datasets become “living fixtures”: reusable test cases drawn from real production failures that grow as engineers review and add new Signal findings. Preserve new variants alongside successful cases whose behavior you want to maintain, so future experiments reflect what the agent encounters in production. Keep the examples and evaluation criteria fixed within each comparison of the baseline and candidate.
The handoff from a Signal finding to a regression dataset still requires manual work. Engineers select and add examples, define expected behavior, and configure the evaluators and experiments. The team remains responsible for acceptance thresholds and the decision to ship.
A concrete example: false completion after a tool failure
Imagine a support agent that must update an account record and summarize the result. In several runs, the update tool returns a retryable error. The agent repeats the call without changing its arguments, exhausts the retry budget, and then writes a polished summary saying the record was updated.
A final-answer evaluator may pass the response because it contains the expected account details and reads like a successful resolution. Infrastructure monitoring may also stay green because the application handled the request and returned normally.
Signal can identify the repeated sequence across traces: tool failure, unchanged retry, no verified state change, success claim. It can group those runs into one issue, attach the relevant traces, and point the investigation toward retry and completion logic rather than the wording of the final answer.
Once the team confirms the issue, it can turn the finding into:
- A deterministic evaluator that checks whether each success claim is supported by a verified record change.
- A regression dataset built by an engineer adding representative examples from Signal’s supporting traces, including the original request, relevant tool-error conditions, and expected recovery behavior.
- A harness change that treats the tool result as unresolved and changes the stopping condition.
- An experiment that compares baseline and candidate on those cases, including successful update flows, to check whether the fix resolves the failure without breaking working behavior.
For a case where retries are exhausted, the expected behavior might be to report that the update failed and follow the defined escalation path. The team preserves that case for future regression tests and adds new variants as production reveals them. Signal keeps watching for recurrence and for failures the suite does not cover.
From investigation to a reviewed change
For Enterprise workflows, Signal can lead into repo-aware Managed Agents. With approved access, a managed agent can inspect trace evidence, gather repository context, create evaluation artifacts, and propose a branch or pull request. Engineers review, modify, approve, or reject the work before deployment.
That boundary matters. A production failure should be easier to investigate, but the system should not make uncontrolled changes to its own behavior. The safe loop separates evidence, proposal, verification, and deployment.
The verification step also has to cover the complete agent system. Agent experiments compare outputs, tool use, retrieval quality, latency, trajectories, and eval results across baseline and candidate runs. That is more useful than proving a prompt improved one score while the new harness became slower, more expensive, or less reliable elsewhere.
The harness is in scope
Signal makes a broader point about agent reliability: the model is only one possible source of failure. The prompt, context assembly, retrieval, tool definitions, permissions, memory, routing, retry logic, stopping conditions, and subagent coordination all shape the trajectory.
A strong model can still perform badly when the harness provides stale context, an ambiguous tool schema, or a retry policy that repeats work without learning from the previous result. A model upgrade can improve final-answer quality while increasing tool calls and timeout risk. The seven-model harness experiment showed exactly that pattern: output quality moved less than operational behavior.
Signal gives the team a way to discover where those system-level problems recur in production. The resulting issue may require a model change, but it may just as plausibly require a retrieval fix, a tool change, a routing rule, a state check, or a better stopping condition.
What changes when telemetry has an agent reader
Detailed trace investigation has traditionally required a human reader. Engineers inspect traces, review eval results, form a theory, and decide what to investigate. Signal adds an automated investigation layer that can repeatedly examine trajectories, connect evidence across runs, and return a prioritized issue.
That’s the practical bridge from observability to continual improvement:
- Traces provide evidence
- Signal discovers the recurring problem
- Engineers add confirmed failures to datasets and define evaluators that make those cases reusable regression tests.
- Managed Agents or engineers propose a change
- Experiments verify whether the complete system improved.
This is also a controlled version of a self-improving agent system. The agent does not rewrite and deploy itself without oversight. Production behavior feeds a governed loop in which automated workers investigate and propose while engineers remain accountable for policy, review, and release.
Let your production environment inform your eval suite
A fixed eval suite cannot be assumed to cover the full production state space. Signal closes part of that gap by finding recurring failures the team did not think to encode.
Trace the work, let Signal find recurring patterns, and carry confirmed findings into datasets and evaluators you maintain over time. Those cases give the team a way to check whether a fix holds as the agent changes, while engineers remain in control of production releases.
To see the workflow in the product, read How to debug production AI agents with Signal. For the broader architecture, read Building the AI factory for self-improving agents.
FAQs
What is Arize Signal?
Signal is an always-on AI worker in Arize AX that continuously reviews production agent traces, remembers known issues, discovers recurring and emerging failure patterns, and groups related traces into prioritized investigation reports with evidence and recommended next steps.
How is Signal different from a monitor or predefined eval?
A monitor or predefined evaluator measures a condition the team already knows how to express. Signal searches production trajectories for recurring patterns the team may not have named yet. The two work together: Signal discovers a failure, and a targeted evaluator or monitor protects against that failure after the expected behavior is defined.
How is Signal different from Agent-as-a-Judge?
Agent-as-a-Judge is an evaluation method for complex, multi-step criteria. It inspects trace context at runtime and produces labels or scores. Signal is the continuous investigation layer across production traffic: it identifies which recurring and emerging patterns deserve attention, groups the evidence, and opens an issue. Signal findings can lead to Agent-as-a-Judge criteria when the behavior remains too complex for a fixed rubric or code check.
Does Signal automatically change production agents?
No. Signal discovers and investigates issues. Managed Agents can gather more context and propose artifacts or code changes when they have approved access, but engineers review, modify, approve, or reject those changes before deployment.
What should a team do after Signal finds an issue?
Confirm the failure and define the expected behavior, then preserve representative examples from the supporting traces in a dataset. Pair those cases with an evaluator that catches the issue and test the baseline and candidate on the same fixed examples before release. Engineers curate and add the cases manually, growing the dataset into a living fixture as production reveals new variants. Keep Signal watching for recurrence and for failure patterns the suite does not cover.
Does Signal replace manual trace review?
Signal reduces the need to search large volumes of traces manually. Engineers should still inspect the evidence behind important issues, especially before changing evaluators, prompts, tools, routing, permissions, or production behavior.
Additional resources
- How to debug production AI agents with Signal: the product walkthrough
- AI agent tracing and evaluation: how to instrument, inspect, and evaluate trajectories
- AI agent testing: the failure catalog for production agents
- How to evaluate AI agents: datasets, experiments, regression cases, and release gates
- Building the AI factory for self-improving agents: Signal, Managed Agents, Agent-as-a-Judge, and experiments
