Follow along with code: This guide has a companion notebook with runnable code examples. Find it here.
Tutorial Structure
Do the parts in this order. Each part uses the result of the part before it.Identify & Edit Prompts
Goal: Find and fix a wrong classification in a trace.You find the failing span, replay it in the Playground, and edit the prompt.
You save the original prompt and the edited prompt as two versions.
Test Prompts at Scale
Goal: Measure the prompt on a labeled dataset.You run the prompt on every example in the dataset and get an accuracy score.
You use an LLM evaluator to explain why each wrong output failed.
Compare Prompt Versions
Goal: Measure whether your edits improve the prompt.You create one version with a new instruction and one version with new model parameters.
You run both versions on the dataset and compare the results.
Optimize Prompts Automatically
Goal: Let an algorithm improve the prompt for you.You give your experiment results to Prompt Learning, which generates a new prompt version.
You run the new version on the dataset and measure the improvement.
What You’ll Have by the End
When you complete the tutorial, you have:- A dataset and an experiment workflow to test prompts.
- Multiple prompt versions that Phoenix tracks.
- Evaluation feedback that explains each failure.
- An automated optimization loop that uses Prompt Learning.

