Skip to main content
Skip to prompt-learning-for-classification if you want to see the notebook.

​
What is Prompt Learning?

Prompt Learning is an algorithm developed by Arize to optimize prompts based on data. See our detailed blog on Prompt Learning, and/or a quick summary of the algorithm below.
The pipeline, which uses Phoenix extensively, works as follows:
  • Upload a dataset of inputs/queries to Phoenix
  • Run a Phoenix experiment on the dataset with your unoptimized, base prompt
  • Build LLM evals with Phoenix or human annotations to return natural language feedback
    • e.g. explanations -> why this output was correct/incorrect (most powerful)
    • e.g. confusion reason -> why the model may have been confused
    • e.g. improvement suggestions -> where the prompt should be improved based on this input/output pair
  • Use meta-prompting to optimize the original prompt
    • feed prompt + inputs + outputs + evals + annotations to another LLM
    • ask it to generate an optimized prompt!
  • Run and evaluate new, optimized prompt with another Phoenix experiment

​
Prompt Learning for Classification

In this cookbook we use Prompt Learning to improve accuracy of GPT-4o-mini on classification of support queries. To view and run the notebook, first clone the Prompt Learning repository.
Navigate to notebooks -> phoenix_support_query_classification.ipynb. You can see the notebook here. But keep in mind you will have to clone the repository and run the notebook within the notebooks folder for the notebook to run!

​
Example Support Queries (our Dataset)

Our dataset contains 154 synthetically generated support queries, each mapping to one of the 30 classes (also synthetically generated).
About one third of the queries/classes are chosen to be inherently ambiguous and not straightforward for GPT-4o-mini to solve, as we want to show progression of accuracy through Prompt Learning. For example:

​
Base Prompt

Below is the base, unoptimized prompt we start off with.

​
Evaluator

In Prompt Learning, your evals/annotations really make or break the optimization. Good evals allow the meta prompt LLM to figure out what changes/improvements are needed to optimize the prompt. Bad evals, such as just correct/incorrect labels, don’t actually guide the meta prompt LLM to making effective prompt updates. We build a complex evaluator as feedback for the prompt optimizer. Specifically, we use LLM-as-judge to return the following eval types:

​
Results - Accuracy

Here we see strong results in one just loop of optimization, and stronger results in 5 loops. This is characteristic of Prompt Learning -> its both data efficient and epoch efficient, allowing you to achieve strong results quickly!

​
Results - New Prompt

In our optimized prompt, Prompt Learning added:
  • descriptions for the classes
  • general rules (which generalize outside of the provided dataset)
  • common decision pivots
  • few shot guidance
  • much better prompt to the human eye
Keep in mind the new prompt is not deterministic. We are using an LLM to generate optimized prompts at every epoch and therefore the new prompts (and their accuracies) will not be the same, but you should consistently see improvements based on our testing.