Skip to main content
When you change the agent, you need to check whether its answers improve without breaking requests it already handles well. You’ll save reviewed requests handled by the Wonder Toys agent, along with the expected behavior, in a dataset, a collection of cases you can reuse to test changes.
Failed, passing, and difficult requests become test cases with an input, expected behavior, and an optional source span, ready to test future changes.Failed, passing, and difficult requests become test cases with an input, expected behavior, and an optional source span, ready to test future changes.

​
Before you start

Use the requests you reviewed in Investigate and label a failed run. Start with the $25, $35, and $1 shopping requests, using the outcomes you actually observed. You’ll run these requests again to test changes, so each one needs enough context on its own. The shopping requests above include the shopper’s requirements; a follow-up question would also need the earlier conversation.
Open PXI and share links to the traces you reviewed. Keep Manual Approval enabled and review the proposed cases before saving them.
PXI proposing cases for the reviewed 35 dollar, 25 dollar, and 1 dollar requests, with expected behavior and links to the source spans

PXI's revised proposals after reviewing the budgets and source spans. These proposals haven't been saved.

​
Step 1: Choose cases that test different conditions

Choose requests that reflect the situations your users encounter. Include:
  • Failures you want to fix. For example, the recorded $25 request returned an over-budget recommendation.
  • Successful requests you want to keep working. The recorded $35 request returned products below its limit.
  • Difficult conditions the agent should handle. The $1 request checks what happens when no product meets the budget.
Three reviewed Wonder Toys requests with different budgets: the 35 dollar request has a passing budget score, while the 1 dollar and 25 dollar requests have failing scores

An AI assistant added these budget reviews: 1 means pass and 0 means fail.

If all your runs passed, keep them so you can check that future changes still handle them correctly. These requests are a starting point for testing budget behavior. As you review more traffic, add other needs and conditions your users encounter, such as different ages or product types.

​
Step 2: Add a request to a dataset

Use the top-level span because this dataset will test the complete shopping request. A ChatAnthropic or search-products span would give you the input and output of just that step.
  • Open a reviewed trace and select Wonder Toys at the top of the tree.
  • Choose Add to Dataset, then select wonder-toys-shopping if it already exists.
  • To create it, select the + beside the dataset selector, enter wonder-toys-shopping, and select Create Dataset.
The top-level Wonder Toys span selected in Phoenix, with its complete shopping request and the Add to Dataset icon outlined in teal

Select the top-level Wonder Toys span, then use the highlighted database icon to add it to a dataset.

​
Step 3: Set the expected behavior

Phoenix fills the editor from the span. Check these fields before saving:
  • Input: Keep the shopper’s complete request in the input field.
  • Output: Replace the captured answer with your expected behavior. This is the reference output you’ll use when assessing future answers.
  • Metadata: Optionally record why you selected the case, such as case_type: budget_failure.
For the $25 request, replace the JSON in Output with:
We use expected_behavior to describe a good response without requiring exact wording. This is a field we chose; saving it doesn’t run an evaluation. Keep it consistent with the request. Editing the example won’t change the original trace.
Phoenix dataset example editor with the shopping request as input and expected behavior as the reference output

Check the request and replace the recorded response with the behavior the test should expect.

Select Add Example, then repeat for the other requests, selecting the same dataset each time. Update each reference to match its request.

​
Step 4: Review the saved cases

Open Datasets & Experiments → wonder-toys-shopping → Examples. Select an example’s ID to read its full input and reference output, then check that:
  • Each input contains enough information to run the request again.
  • Each reference describes the expected behavior rather than copying a failed answer or its pass/fail label.
  • Cases cover different conditions, including passing requests and any failures you observed.
Wonder Toys shopping dataset with the 25 dollar, 35 dollar, and 1 dollar requests and their reference outputs

The dataset keeps the reviewed requests and their expectations together for future experiments.

In an example added this way, View Source Span takes you back to the original evidence. Phoenix versions dataset changes, letting you identify which cases an experiment used.
Review expectations when product data or requirements change. Cases involving orders also need a known starting state, so a previous test’s purchase or cancellation doesn’t change the next result.Later, dataset splits can group cases for development, validation, or focused checks. You don’t need to create splits for this first dataset.
Your dataset now holds the requests and expectations for future tests. Next, choose what to evaluate to define how you’ll check the agent’s answers against those expectations.