How to debug production AI agents with Signal in Arize AX

Signal is a managed agent built into Arize AX that reviews your production traces on a recurring schedule, identifies recurring failure patterns, and turns them into prioritized issues with supporting evidence, a likely cause, and recommended next steps.

Self-improving agents do not rewrite and deploy themselves unchecked. They operate within a controlled loop: production behavior becomes evidence, evidence becomes an investigation, and proposed changes are tested before an engineer decides what ships.

Signal is the investigation layer in that loop. Built into Arize AX, it continuously reviews production traces, groups recurring failures into prioritized issues, and surfaces the supporting evidence, likely cause, and recommended next step. With repository access, Managed Agents can carry the investigation into the codebase and propose a pull request. Datasets, evaluations, and experiments then determine whether the change actually improved the agent.

Availability: Signal issue detection is available across Arize AX plans. Repository-backed investigation, pull-request creation, Agent Studio, and broader managed-agent workflows are Enterprise capabilities. For more information, check out the launch blog post.

TL;DR: What Signal does

• Signal reviews production traces on a recurring schedule and groups related failures into ranked issues.
• Each issue includes an investigation, supporting traces, and a suggested prompt, code, configuration, or evaluation change.
• Teams can connect a GitHub repository so that, when you choose to act on an issue, Signal can investigate the implementation and open a pull request for review
• Matching traces can become a regression dataset, while evaluators and experiments determine whether the proposed fix actually improved the agent.
• Developers remain responsible for reviewing and merging changes. Managed agents do not deploy directly to production.

Why production AI agents are difficult to debug

Coding agents have compressed the time required to implement a change. The difficult part increasingly happens before the code is written: finding the relevant production behavior, recognizing the recurring failure, understanding its likely cause, and deciding what should change. Agent failures also rarely look like conventional software failures. Many of these patterns map to common AI agent failures that only become visible in traces. A request can return successfully while the agent:

• Routes to the wrong worker
• Calls the wrong tool
• Passes malformed arguments
• Retrieves an empty or irrelevant result
• Repeats a planning step without making progress
• Falls back to model memory without telling the user
• Produces a plausible answer through an unreliable trajectory

The codebase defines the paths an agent can take. The trace records the path it actually took. That makes traces the source of truth for agent behavior. But traces alone do not solve the scale problem. A developer can inspect one failed run manually. Reviewing thousands or millions of runs for related semantic failures is a different task. Evaluations help identify which runs violated an application’s quality criteria. They might show, for example, that hundreds of runs selected the wrong agent or answered without grounded evidence. But a failed evaluation does not necessarily explain whether those runs share one routing defect, several unrelated tool failures, or a problem with session state. Signal performs that investigation continuously.

Tutorial: How to use Signal to debug a production AI agent

Step 1: Instrument the full agent execution path

Signal depends on production evidence, so the first requirement is adding instrumentation to your agent so that the agent telemetry can flow into Arize AX. A useful trace should capture more than the final model response. It should make the agent’s trajectory legible, including:

• User input and final output
• LLM calls
• Latency and token counts
• Tool names, inputs, and outputs
• Retrieval operations
• Agent or node transitions
• Exceptions and error states
• Retries and repeated steps
• Session or user metadata required for segmentation

Arize supports instrumentation through a coding agent, as well as manual instrumentation. For the coding-agent workflow, install the Arize instrumentation skill:

npx skills add Arize-ai/arize-skills –skill “arize-instrumentation” –yes

The coding agent can inspect the application, add the relevant OpenInference instrumentation, and help verify that traces are reaching the project. Before enabling Signal, inspect at least one representative trace manually. Confirm that a developer could answer these questions from the trace alone:

1. Which route did the agent select?
2. Which tools did it call?
3. What did each tool return?
4. Where did the trajectory diverge from the expected behavior?
5. What did the user ultimately receive?
6.  

Missing tool outputs or flattened agent steps will limit the quality of any downstream investigation.

Step 2: Add evaluations for important behavioral contracts

Signal can begin working once traces are present. Evaluations make the evidence more precise by encoding the behaviors your application promises to users. For related guidance on building trustworthy agent scorecards, see AI agent evaluation and how to improve agent skills with tracing and evals. Consider a financial research agent that answers questions involving current market data. One behavioral contract might be:

That contract could become an agent_selection_correctness, tool_call_required, or groundedness evaluator. Other useful production evaluations include:

• Correct agent or tool selection
• Successful task completion
• Answer groundedness
• Citation validity
• Maximum retry count
• Session coherence
• Policy compliance
• Format adherence

Choose evaluations that correspond to user-visible or operational contracts. A generic “response quality” score may reveal that something went wrong, while a routing or tool-use evaluator gives the investigation a much sharper starting point.

Step 3: Enable Signal on the tracing project

Once traces are flowing:

1. Open the relevant tracing project in Arize AX.
2. Select the Signal tab from the project navigation.
3. Click Enable Signal.

Enabling Signal creates a recurring automation for that project. From Configure Signal, teams can adjust the cadence and provide additional context for the investigation. Signal reads traces from the selected project and presents its findings in the same project-level view. No repository connection is required at this stage. A useful adoption path is to begin with trace-only investigations, review the quality of the clusters, and add repository access after the team trusts the evidence.

Step 4: Read each Signal issue as an incident report

The Signal view shows issue volume over time and a ranked list of detected patterns. Opening an issue reveals the investigation and its supporting traces. A useful issue should help you understand:

• The recurring behavior: What is the agent doing incorrectly?
• The affected population: Which traces or sessions show the same problem?
• The likely impact: Does the failure affect correctness, reliability, cost, latency, or user experience?
• The evidence: Which spans support the diagnosis?
• The proposed intervention: Should the team change a prompt, tool definition, state transition, evaluator, or code path?

Treat severity as a triage signal rather than an automatic merge decision. Frequency matters, although volume alone cannot capture impact. During the webinar, the Arize team showed an internal agent that entered a loop involving repeated completion and to-do updates. The behavior appeared across a relatively small number of traces, but one trace repeated the loop 105 times. That pattern created meaningful cost and reliability risk despite its limited trace count. By comparison, a formatting problem might occur hundreds of times while causing little user harm. Review the issue through three lenses:

1. How frequently does it happen?
2. How damaging is each occurrence?
3. How widely can the failure spread?

Step 5: Connect the repository and generate a pull request

When an issue is well supported by the traces, the next step is to give Signal enough context to connect the runtime behavior to the implementation. For supported Enterprise workflows:

1. Open the menu on the Signal tab.
2. Select Configure Signal.
3. Enable Attach Repo.
4. Choose or create a GitHub integration.
5. Select the repository from your connected GitHub App installation (shown as owner/repo).
6. Save the configuration.

Signal can then use the repository to investigate the relevant code and open pull requests for proposed fixes. The pull request still requires your review using the same standards you would apply to a human-authored change:

• Does the diff address the root cause shown in the traces?
• Is the change narrow enough to evaluate?
• Does it introduce unrelated refactoring?
• Are important invariants enforced in code?
• Are prompt instructions being used where deterministic controls would be safer?
• Does the change include or suggest an appropriate regression test?
• Could it affect latency, token use, or tool-call volume?
• Is there a clear rollback path?

Managed workers create artifacts for review. They do not merge or deploy the change on their own.

Step 6: Turn the production failure into a regression dataset

A pull request addresses the current defect. A dataset prevents the team from forgetting the behavior that exposed it. Save representative traces from the issue as dataset examples. Preserve enough metadata to explain both the input and the expected trajectory:

Include variation within the failure mode. For a retrieval problem, the dataset might contain blocked pages, empty pages, irrelevant results, timeouts, and successful pages. This helps determine whether the fix generalizes beyond the exact trace that inspired it. A useful division of labor is:

When a failure is important enough to change production code, it is usually important enough to preserve as a regression case.

Step 7: Compare the baseline and candidate before merging

Run the same dataset against both the current production configuration and the proposed configuration. A basic experiment should compare:

• Evaluator pass rate
• Task completion
• Tool selection
• Groundedness
• Latency
• Token usage
• Number of tool calls
• New failure modes introduced by the candidate

In Arize AX, agent experiments can run a dataset against a registered agent endpoint and compare multiple runs. The comparison view includes row-level outputs, evaluator deltas, latency, token counts, and differences in tool-call paths when tracing is enabled. The safest sequence is:

1. Run the dataset against the current production agent.
2. Run the same dataset against the proposed change.
3. Compare the evaluator results and operational metrics.
4. Inspect every regression, even when the aggregate score improves.
5. Merge only after the candidate passes the team’s release criteria.

A correct result on one previously failing example is weak evidence. Consistent improvement across a representative dataset is much stronger.

Worked example: When a research agent answers from memory

Consider an agent responsible for answering questions about current companies and executives. The intended path is straightforward:

1. The user asks a time-sensitive question.
2. The agent calls a web research tool.
3. The tool retrieves a relevant page.
4. The agent grounds its answer in the retrieved content.

In one production failure from the webinar, the research tool returned a JavaScript-gated page with little usable text. The agent then skipped another retrieval attempt and answered from model memory. The final answer happened to be factually correct. The trajectory was still unreliable. For a different company, a recent leadership change, or a more consequential question, the same fallback could produce a stale or fabricated answer. An agent-selection evaluator marked the traces as incorrect. Signal then grouped several instances into one recurring issue: the research agent was answering from memory after unsuccessful retrieval. Instead of leaving the team with a long list of failed labels, the issue included matching traces and a proposed change. A candidate policy might look like this:

For time-sensitive questions:
1. Call an approved live-data or web-research tool.
2. When a source is blocked or empty, try an alternative source.
3. Categorize retrieval failures explicitly.
4. Report unsuccessful retrieval rather than guessing.
5. Ground factual claims in content returned by the tools.

Depending on the application, some of those controls may belong in the system prompt. Others deserve code-level enforcement. For example:

• A maximum retry count should usually be deterministic.
• Required citations can be validated after generation.
• A “no answer from memory” policy can be enforced by checking whether an approved tool succeeded.
• Retrieval failures can be represented as typed states rather than free-form model decisions.

Signal’s proposed fix should be treated as an engineering hypothesis. The trace evidence, dataset, and experiment determine whether that hypothesis survives contact with the system.

Common Signal use cases for production agents

Signal is especially useful for failures that recur across traces but remain difficult to identify through dashboards alone.

Signal can also reveal where the AI trace is only part of the story. An agent may complete its LLM and tool steps correctly while a UI component, downstream service, or application state update fails. For investigations that need additional operational context, managed workers can use configured integrations such as GitHub, Datadog, or custom skills alongside the Arize traces.

When to use Agent Studio instead of Signal

Signal is purpose-built for recurring investigation of production traces. Agent Studio is useful when the team needs a custom worker with a narrower task, a different trigger, or additional systems. Examples include:

• A daily cost analysis
• A weekly agent health report
• Investigation of traces that cross a custom metric threshold
• Root-cause analysis triggered by a monitor
• Dataset generation from a specific class of failures
• A targeted repository fix
• Correlation between AI traces and APM data

Teams can start Agent Studio from a template or a blank task, bind the worker to a tracing project, select a preset, attach the required skills, and run it as either a one-off session or an automation. Automations can run on a recurring schedule or a metric threshold. A practical investigation prompt might be: Review production traces from the last 24 hours. Find recurring failure modes affecting at least three sessions. For each cluster:

• Cite the supporting trace IDs and provide a reusable filter.
• Describe the user or operational impact.
• Identify the most likely root cause.
• Recommend the smallest safe prompt, code, or configuration change.
• Specify the regression examples and evaluators needed to test the change.
• Estimate possible effects on quality, latency, token use, and tool-call volume.

Then, rank the clusters by impact, frequency, and cost. Do not open a pull request. Return an investigation report for review. Beginning with a report-only workflow makes it easier to evaluate the worker’s reasoning. Once the clusters and recommendations are consistently useful, the team can connect a repository and allow the workflow to produce candidate branches or pull requests.

How Signal fits into a self-improving agent loop

In practice, a self-improving agent is a closed engineering loop with explicit review gates. That loop is the same shift described in from human-operated agent development to systematic agent improvement, and in the broader AI factory for self-improving agents in Arize AX. Production behavior becomes evidence. Evidence produces a candidate change. Evaluations determine whether that change deserves to ship. Signal provides the investigation layer in this loop: It connects what the agent did in production with what the engineering team should examine next. When managed workers need additional context, they can operate inside an isolated sandbox with access to a bound Arize project, an optional repository, and configured external tools. Their outputs can include investigations, evaluator labels, branches, and pull requests. Production deployment remains outside the worker’s control.

Best practices for getting useful Signal investigations

• Trace decisions, not only model calls. A collection of isolated prompts and completions will rarely explain an agent failure. Capture routing, tools, retrieval, state changes, and parent-child relationships.
• Write evaluations around behavioral contracts. “Good response” is difficult to act on. “Used an approved live-data source for a time-sensitive question” gives Signal and the developer a specific failure to investigate.
• Prefer the smallest testable change. Broad prompt rewrites and large refactors make it difficult to determine why a metric moved. A narrow change produces cleaner experimental evidence.
• Convert meaningful failures into durable assets. Important production failures should leave behind a dataset example, evaluator, test, or monitor. Otherwise, the team may rediscover the same behavior months later.
• Measure efficiency alongside quality. A candidate can improve correctness while doubling tool calls or latency. Compare task quality, cost, and execution behavior before shipping.
• Keep humans at the change-control boundary. The managed worker can perform the time-consuming investigation and draft the change. The developer remains responsible for validating the cause, evaluating the candidate, and deciding what reaches production.

As the webinar framed it, the engineering role shifts from first responder to reviewer.

From observability to action

For years, observability workflows ended with a diagnosis. The system reported what happened, and a human carried the evidence through every remaining step.

Signal extends that workflow into investigation and action.

Traces establish what the agent did. Evaluations establish whether the behavior met the application’s standards. Signal finds the recurring patterns that deserve attention. Repositories provide implementation context. Datasets and experiments determine whether a proposed change earned the right to ship.

Start with an active project that already contains collected telemetry data to expose recurring behavior. Enable Signal, review the first clusters, and compare them with the traces. Connect a repository after the investigations prove useful. Preserve the failures as a dataset, run the candidate against the baseline, and merge only when the evidence shows a real improvement.

Signal issue detection is available across Arize AX plans, while repository-backed pull requests and managed agents are Enterprise capabilities. Take a look at the documentation for further information.