AI agent regression testing with Agent Experiments in Arize AX

Key takeaways

• Fixing one agent failure can introduce another. A change that improves one workflow can still affect routing, tool use, or task completion elsewhere in the agent. In our demo run, safety improved, but average task completion fell from 0.89 to 0.72.
• Your regression dataset needs working cases, too. Include common workflows, known failures, and edge cases so testing catches side effects beyond the problem you set out to fix. That mix is the same idea behind building a dataset in Arize AX and the regression evals that protect customer-critical behavior.
• Compare the current agent against one proposed change. Run baseline and candidate configurations against the same dataset in Arize AX to see how the change affects the scenarios you depend on.
• Measure action safety and task completion separately. An agent that avoids unsafe actions can still leave valid requests unfinished. Both evaluations matter when deciding whether to ship. See our agent evaluation handbook for how those signals differ.
• Investigate the behavior behind the scores. Compare individual results and open their traces to inspect what changed. When a candidate introduces regressions, refine the fix and rerun the tests before release.

A prompt fix can pass the test that motivated it and still make your agent worse.

This pattern applies to support agents, coding agents, research agents, shopping agents, and internal workflow agents: a fix in one part of the agent can change routing, tool use, or completion behavior somewhere else.

That was the focus of our recent AI builders session I held with Fuad Ali, a product manager at Arize focused on agent experimentation. In this session, we used a demo where an e-commerce agent cancelled an order without asking for an order ID or confirmation. The proposed fix initially seemed straightforward: require the necessary details and explicit confirmation before taking actions such as purchases or cancellations.

But things quickly got a bit more complicated.

While the updated agent handled the cancellation request more carefully, a previously working purchase scenario stalled. Even with the product and shipping details supplied, the agent returned product information without completing the purchase or asking for confirmation. Across the test dataset, safety improved while the average task-completion score fell from 0.89 to 0.72.

Testing only the original failure would have made that change look ready to ship. AI agent regression testing checks the workflows around it, giving you evidence of whether a proposed fix preserves the behavior customers already depend on.

This tutorial shows how to run that comparison with agent experiments in Arize AX. You’ll connect a deployed agent, run its current and proposed configurations against the same dataset, and inspect evaluation scores and execution traces to understand what improved, what broke, and what needs another iteration before release.

What is AI agent regression testing?

AI agent regression testing checks whether a change breaks behavior that previously worked. For an agent, that means checking how a prompt, model, tool, or policy change affects the workflows customers depend on.

An Agent Experiment runs a dataset through the full deployed agent. When tracing is configured, each result can link back to the LLM calls, tool calls, retrieval steps, and orchestration path behind the response. In Arize AX, experiment results bring together the output, evaluation scores, and the trace of what the agent did.

You can reuse instrumentation and evaluators across production and experimentation, then compare results side by side.

A cancellation request and a purchase request both involve tools that change order state. An instruction intended to make those actions safer can also change whether the agent completes them. Testing the cancellation case alone would answer whether the agent stopped cancelling too quickly. But testing its other workflows reveals whether the proposed fix creates a different problem.

How to run AI agent experiments in Arize AX

The walkthrough starts with an existing agent that is already deployed and instrumented for tracing. Arize AX calls that application through its endpoint; the agent continues to run in its own environment.

1. Connect the agent to Arize AX

First, open Remote Agents and create a configuration for your application. Enter the agent’s name and endpoint URL, then add the headers AX needs to authenticate its requests.

Next, define the input schema to match the request body your endpoint accepts. For Wonder Toys, the required message field contains the customer’s request. The request also includes a customer ID because the tools need access to customer state.

This configuration gives AX a way to send dataset inputs through the running application. You can then test the agent’s tool use and responses without rebuilding its execution logic in a separate test script.

Check the tracing connection as part of the setup. AX sends trace context with experiment requests, and your application needs to propagate it so the agent’s spans link back to the correct run. That lets you open a failed result and inspect the model and tool calls behind it.

2. Create a dataset that includes the workflows you need to preserve

In Datasets, create or upload your test cases. Each row should contain a customer request and the context needed to run it. For any agent, the dataset should cover common paths, known failures, edge cases, and workflows where the agent takes action.

In the Wonder Toys demo, that meant message, customer ID, workflow, scenario ID, expected behavior, and expected tools. The scenario ID gives each case a stable name; the workflow identifies which part of the application it exercises. Expected behavior and tools describe what a successful run should look like.

It’s important to be specific about the expected outcome. For the purchase case, the customer had supplied product and shipping details, so the expectation was to complete the purchase and return an order ID. A response describing the product would leave that task unfinished.

Include the cancellation failure, but keep the working purchase case and the agent’s other common workflows in the dataset. You’ll need those examples to check whether the new instruction has side effects.

The demo’s dataset covers search, product details, purchases, order status, and cancellations. That coverage is how we found the checkout problem.

3. Save the current and proposed configurations as presets

In the remote agent configuration, create two request presets. The baseline uses the agent’s existing instructions. The candidate adds the policy we want to test.

In plain language, that policy requires the necessary details and explicit confirmation before a purchase or cancellation, with one focused follow-up question when something is missing.

Both presets use the same endpoint and will run against the same dataset. The added instruction is the change under test.

Your application needs to know how to apply the configuration in the request. A preset supplies values to the endpoint; the request-handling code determines how those values affect the agent. In this demo, that layer lets us change the instructions without switching to a different endpoint.

4. Run both presets with the same evaluators

Open the dataset and select New Experiment → Run in Agent Playground. Choose your registered agent and the baseline preset. Check the body template before running. AX fills template placeholders from each dataset row, so {{dataset.message}} maps the row’s message value into the request. Map the remaining fields according to your endpoint’s schema.

{
  "message": "{{dataset.message}}",
  "customer_id": "{{dataset.customer_id}}",
  "workflow": "{{dataset.workflow}}",
  "expected_behavior": "{{dataset.expected_behavior}}"
}

AX hydrates each {{dataset.column_name}} placeholder from the current dataset row, so every row becomes one request to the agent.

For this experiment, we use two evaluators:

1. Action safety checks whether the agent takes unsafe tool actions. For the cancellation case, we want it to obtain the required information and confirmation before proceeding.
2. Task completion checks whether the agent finishes the expected workflow. For the purchase case, we want to know whether the customer actually gets an order placed.

These measure different things. Avoiding an unsafe action is useful, but we also need to know whether the agent can still carry out valid requests.

Run the baseline, then repeat with the candidate preset, keeping the dataset and evaluators unchanged. Each row produces an experiment result that you can compare across the two configurations.

5. Compare the cancellation and purchase results

Once both runs are available, open the dataset’s Experiments tab, select the runs, and click Compare. The comparison view shows the outputs and evaluation results side by side.

The aggregate results give us a reason to investigate: action safety improved, but average task completion dropped from 0.89 to 0.72.

Start with the cancellation case that motivated the change. The baseline cancelled the recent order immediately. The candidate responded: “I can do that. What’s the order ID for your most recent order? If you don’t have it, I can look it up.” That is the behavior we wanted for this request. The agent asks for missing information before proceeding.

Now open the purchase case. The baseline placed the order using the product and shipping details the customer supplied. The candidate returned product information without completing the purchase. It also failed to ask for confirmation, which would have given the customer a way to continue.

The evaluator explanation identifies the purchase failure clearly: the response did not satisfy the expected behavior, even though the necessary details were present.

From the result, open the experiment trace to compare the prompts, model responses, and tool calls. This is where you can investigate how execution changed rather than trying to diagnose it from the final answer alone.

Also check whether you agree with the evaluator. AX supports annotations in the comparison view, so you can flag a result when the behavior looks correct but the scoring criteria need work.

Test your next agent change with Arize AX

Start with a failure you’ve seen in production and the working scenarios you need to protect. With Arize AX, you can compare your current agent against a proposed change, then inspect evaluation results and traces to see what’s ready to ship and what needs another iteration.

Once the comparison is repeatable in the UI, you can run the same experiment pattern as a CI regression gate. Watch the full webinar.