Skip to main content
In Part 3, you fixed one error type by hand. Six other error types remain. To fix each one by hand takes much time. In this part, you use the Prompt Learning SDK to generate a new prompt version from your experiment results. At the end, you have Version 5 of the support-classifier prompt and you know its accuracy on the dataset.
Follow along with code: This guide has a companion notebook with runnable code examples. Find it here, and go to Part 4: Optimize Prompts Automatically.

What is Prompt Learning?

Prompt Learning uses the feedback from your evaluations to write a better prompt. An LLM reads the original prompt, the outputs, and the feedback. Then it writes a new prompt.
Prompt Learning workflow
To learn how Prompt Learning works, see these resources:

Install the Prompt Learning SDK

Install the Prompt Learning SDK.

Load Experiment for Training

The Version 4 experiment is the training data. Prompt Learning uses its outputs and evaluations to write the new prompt. Open the Version 4 experiment and copy the experiment ID.
Experiment ID location
The process_experiment function is in the notebook. Copy it from there.

Load Unoptimized Prompt

Load the current prompt from Phoenix. Set PROMPT_VERSION_ID to the ID of Version 4.

Optimize Prompt (Version 5)

Run the optimizer. Then save the optimized prompt to Phoenix as a new version.

Measure New Prompt Version’s Performance

Run an experiment on the support query dataset with the optimized prompt. This code uses support_query_dataset and the evaluators from Test Prompts at Scale. It uses create_task from Compare Prompt Versions.
Optimized prompt experiment results
The accuracy of the optimized prompt is 82%.

Summary

You completed the Phoenix Prompts tutorial.
You started with one wrong classification. Now you have an optimized prompt with 82% accuracy, up from 53%.
You did these tasks:
  • Identified and edited a prompt from a trace to fix a wrong classification.
  • Tested the prompt on a dataset to measure accuracy and find the most frequent errors.
  • Compared prompt versions to measure the effect of a new instruction, new parameters, and a new model.
  • Optimized the prompt automatically with Prompt Learning, from the feedback of your evaluations.
  • Tracked each version of the prompt and each experiment in Phoenix.

Next Steps

For more prompt tutorials, see: