Skip to main content
In one recorded run, a shopper asked for toys under $25, but the Wonder Toys shopping assistant agent recommended a $27.99 kit. Could a clearer budget instruction help it follow the shopper’s limit? In Phoenix Playground, you can edit a model’s instructions and compare its responses. You’ll try a clearer budget instruction on the same model input to see whether it’s worth testing with the complete agent.
Replay one model step with the captured messages and search results. Compare the original prompt with a budget instruction, then test the full agent.Replay one model step with the captured messages and search results. Compare the original prompt with a budget instruction, then test the full agent.

​
Before you start

Open a Wonder Toys trace from Read model, retrieval, and tool steps. Use a run where the model has already received search results. Your answer may differ from the recorded example.

​
Step 1: Load a model call

Start with the model call that received the search results:
  • In your trace, select the ChatAnthropic span after search-products.
  • Open Input and check that it includes the returned products.
  • At the top right of the selected span, use the play button labeled Prompt Playground to open that model call in Playground.
Light-mode animation highlighting the Prompt Playground button on a selected ChatAnthropic span and opening the captured model call in Playground.

The pointer shows the Playground button on the selected model span, then the captured call opening in Playground.

Once Playground opens:
  • Check that the system message, shopper request, tool result, model, and tools are present.
  • Use the API Keys menu to configure your model provider. The Wonder Toys app’s key isn’t automatically shared with Playground.
The output initially shows the recorded answer. You’ll run a new comparison in Step 3.

​
Step 2: Add one prompt change

For this exercise, test whether an explicit budget instruction changes the response:
  • Click Compare to duplicate the prompt.
  • Keep A unchanged.
  • Add the instruction below at the end of B’s system message.
Light-mode animation highlighting Compare, creating prompt B beside A, and pointing to B's System message editor.

Click Compare to create B beside A, then edit B's System message on the right.

Leave the model, parameters, shopper request, and search results the same. This helps you assess the instruction without mixing it with other changes.

​
Step 3: Run and compare the responses

Click Run to get a new response from both versions. A uses the original instructions; B includes your added budget rule. Compare their responses to the same shopper request and search results:
  • Does it recommend a product outside the limit?
  • If no product fits, does it explain that clearly?
  • Is it answering the shopper or requesting another tool call?
Phoenix Playground comparing the original Wonder Toys prompt with a budget instruction on the same captured model input.

Playground shows the two variants side by side. A replay may return an answer or a request to call a tool.

If an output requests a tool call, you haven’t seen its final answer yet. Playground displays the request but doesn’t execute Wonder Toys’ application tools. You’ll test that complete sequence in the next lesson.

​
Step 4: Decide what to test next

You’ve tested a possible change, so use the responses to decide what to do next:
  • B follows the budget where A didn’t: Keep the instruction as a candidate for a full-agent test.
  • B has the same problem or introduces another: Revise the instruction and compare again before testing it further.
  • B requests another tool call: Test it with the complete agent to see whether that call leads to a suitable answer.
For example, if A recommended an over-budget toy and B ruled it out, the new instruction helped on this request. If B requested another tool call, you still need to see the final answer before deciding whether it helped. A better response here is evidence for one model input. The replay hasn’t updated the running Wonder Toys agent or established that the change helps other requests. If you want to test the instruction further, save it in academy/evaluations/prompts/budget-rule.txt in your local checkout. Then continue to Run a Phoenix dataset experiment, which reads that file and compares both agent versions across the same requests.