Skip to main content
Changing an agent’s model, prompt, or tools can improve some tasks and break others. In Compare results and agent steps, you compared versions on the same shopping requests. For tasks that change application data, a fair comparison also needs the same starting environment and checks of the completed work. Harbor packages instructions, an isolated test environment (a sandbox), and code that checks the outcome (a verifier) into a reusable benchmark task. You’ll start with one Wonder Toys purchase task and inspect its scores and agent trace in Phoenix. You can reuse that task to compare other agent configurations.
Harbor starts a fresh Wonder Toys environment, runs the confirmed purchase request, and verifies saved order state. Phoenix records the rewards and agent trace.Harbor starts a fresh Wonder Toys environment, runs the confirmed purchase request, and verifies saved order state. Phoenix records the rewards and agent trace.

​
Before you start

Use your existing Wonder Toys checkout. You’ll need: Harbor requires Python 3.12 or newer, while the earlier app lessons use Python 3.11. The --python 3.12 option below selects the runtime for the Harbor commands.

​
Step 1: Understand the task

The exercise includes a ready-to-run purchase task. First, read the fictional shopper’s request and the checks below so you know what counts as success:
Buy exactly one Dinosaur Explorer Kit, product toy-051, for me. I confirm the product, quantity, and purchase using my saved payment method. Ship it to Demo Shopper, 123 Example Street, Example City, CA 00000, US. Do not purchase any other products. Tell me the resulting order ID and total.
For this task, the verifier checks that the agent saved exactly 1 order with:
  • The correct items: 1 Dinosaur Explorer Kit, with no extra products.
  • The correct address: The shipping details from the request.
  • The correct owner: academy-shopper, the test shopper’s account.
These checks use the same approach as Write checks for actions and outcomes. A confirmation in the agent’s reply alone won’t pass them.
A Harbor task combines a request, an environment, and checks. The supplied purchase task contains:

​
Step 2: Run the task

Run the supplied task to see whether Wonder Toys meets those checks. Harbor reads the request and verifier from the task files and starts the agent in a fresh environment. From the Wonder Toys repository root, prepare the application files:
Then run Wonder Toys on the shopper’s request:
Use your Phoenix address for --phoenix-url if it differs from the example. Use your app’s .env.local path for --env-file if you configured it elsewhere. These helpers run Harbor with the Phoenix plugin. Once the agent and verifier finish, Phoenix holds the scores and trace you’ll use to explain the outcome.
  • prepare.py copies the existing backend into the Docker build context.
  • run.py calls the Harbor CLI with --plugin arize-phoenix. The agent uses its existing tools and product catalog.
Harbor saves local job files under academy/harbor/jobs/.

​
Step 3: Explain the result

Use the recorded scores to check whether the purchase met the success criteria from Step 1, then follow the trace to understand the agent’s actions. In Phoenix:
  • Open Datasets & Experiments.
  • Select harbor-task/phoenix-academy/purchase-confirmed, the dataset created for this task.
  • Open its Experiments tab and select wonder-toys-purchase-check.
Read the scores in this order:
  • infra_ok: 1 means Harbor recorded no execution exception. If it’s 0, inspect the error before interpreting the task result.
  • reward: 1 means the saved items, address, and owner all matched the requirements; 0 means at least one check failed.
  • Individual checks: ordered_items_match, shipping_address_match, and order_owner_match show which requirement passed (1) or failed (0).
Phoenix verifier scores showing infra_ok, order_owner_match, ordered_items_match, reward, and shipping_address_match, all equal to 1.

Verifier scores from a recorded purchase: the item, address, and owner checks all passed.

A completed task can have reward=0 and infra_ok=1. If an exception occurred, inspect it alongside any rewards before interpreting the result. To explain why the run received that score:
  • Select View run trace, the trace icon above the answer.
  • Look for the purchase-product tool call and inspect its arguments and result. If it’s missing, inspect the model steps to see how the run ended.
  • Compare the product, quantity, address, and shopper account with the checks from Step 1.
Phoenix ATIF trace showing the real Wonder Toys purchase-product call and its result.

The purchase tool received the confirmed cart and returned a successful order. The verifier separately checked the saved order state.

The Phoenix Harbor plugin records the verifier’s scores and converts the agent’s saved trajectory into a trace after the trial. The trajectory uses ATIF, a format for recording agent steps. Phoenix records the outcome; Harbor runs the task and verifier. ATIF timestamps may not provide precise operation durations, so a tool span showing 0ms doesn’t mean the tool took no time.

​
Step 4: Check that the test works

The result you just read depends on the verifier applying those checks correctly. Before relying on it to assess other agent versions, test it with 2 known outcomes. Neither run calls the model. First, run the reference solution. It saves the correct order and should pass:
Then run the do-nothing agent. It leaves no order and should fail:
Open their experiments in the same Phoenix dataset:
  • oracle-purchase-check: Expect reward=1 because the reference solution saved the correct order.
  • nop-purchase-check: Expect reward=0 because no order was placed.
  • Both runs: Expect infra_ok=1, meaning no execution exception was recorded.
If either result differs, investigate the task or verifier before trusting the agent’s score. These checks can catch a test that passes everything or fails even a correct solution. When the reference solution passes and doing nothing fails, you have checked this task against known outcomes. To cover missing consent, cancellations, or stock shortages, add separate tasks with their own success criteria. You’ve now checked recommendations and a saved action. In Decide whether a change is ready, you’ll use your results and the cases you haven’t tested to decide whether to release a change or keep testing.