Skip to main content
In this part, you create two new versions of the support-classifier prompt. Version 3 adds a new instruction. Version 4 also changes the model and its parameters. At the end, you know how much each change improves accuracy.
Follow along with code: This guide has a companion notebook with runnable code examples. Find it here, and go to Part 3: Compare Prompt Versions.

Build Two New Prompt Versions

In Test Prompts at Scale, the prompt had 53% accuracy. Use the results of that analysis to create the new versions.

Edit Prompt Template (Version 3)

The prompt template is the text that goes to the LLM. In Test Prompts at Scale, 30 of 71 errors had the broad_vs_specific error type. This is the instruction that you wrote to fix these errors:
Add the instruction to the prompt and save it as a new version.

Edit Prompt Parameters (Version 4)

A Phoenix prompt also stores the model and its invocation parameters. For more about what a prompt contains, see Prompts Concepts. Create Version 4 with these changes to the model and parameters:

Edit prompt parameters

Compare Prompt Versions

Run an experiment with each new version on the dataset. Then compare the accuracy of the versions.
In the Phoenix UI, open the support-classifier prompt. Copy the version IDs of Version 3 and Version 4.
Open the support query dataset and click the Experiments tab to see the results.
Experiment results comparison
With the new instruction, accuracy is 61%. With the new instruction, new parameters, and the gpt-4.1-mini model, accuracy is 74%.

Summary

You created two prompt versions and measured each one.
  • New instruction only: Accuracy increased from 53% to 61%.
  • New instruction, new parameters, and new model: Accuracy increased to 74%.
To compare prompt variations side by side in the Playground, see Test a prompt.

Next Steps

In Optimize Prompts Automatically, you use Prompt Learning to improve the prompt more. Prompt Learning is an automated prompt optimization algorithm from Arize.