Skip to main content
In Part 1, you fixed one wrong classification. One trace shows one mistake. A dataset shows the patterns across many mistakes. In this part, you run the support-classifier prompt on a dataset of support queries. At the end, you have a baseline accuracy score and a new instruction that fixes the most frequent type of error. You do these tasks:
  1. Upload a dataset of support queries.
  2. Run an experiment with the current prompt.
  3. Score each output and generate feedback with evaluators.
  4. Analyze the results to find the most frequent error type.
Follow along with code: This guide has a companion notebook with runnable code examples. Find it here, and go to Part 2: Test Prompts at Scale.
1

Load Dataset of Inputs

Upload a dataset of support queries. Each row has a query and its correct class.
  1. Download support_queries.csv here.
  2. Go to Datasets and Experiments. Click Create Dataset and upload the file.
  3. For Input keys, select query. This is the input column.
  4. For Output keys, select ground_truth. This is the correct output.
  5. Click Create Dataset. Open the new dataset.

    Upload dataset to Phoenix

2

Run Experiment with Our Current Prompt

Run the current prompt on all the examples in the dataset. The result is your baseline accuracy.

Define Task Function

Define a task function. The task generates an output for each input in the dataset. Here, the task asks the LLM to classify a support query.

Define Evaluators

Define evaluators. An evaluator scores each output and can explain why the output is correct or incorrect.Use two evaluators:
  • ground_truth_evaluator – Checks if the predicted class matches the ground truth.
  • output_evaluator – Uses an LLM to analyze each classification. It returns these fields:
    • explanation – Why the classification is correct or incorrect.
    • confusion_reason – If incorrect, why the model might have made the wrong choice.
    • error_type – If incorrect, what kind of error occurred (broad_vs_specific, keyword_bias, multi_intent_confusion, ambiguous_query, off_topic, paraphrase_gap, or other).
    • evidence_span – The exact phrase in the query that supports the correct classification.
    • prompt_fix_suggestion – An instruction to add to the classifier prompt to prevent this type of error.
The code uses analysis_evaluator_template. Copy the full template from the “Define Evaluators” section of the notebook.

Run Experiment

Run the experiment with the task and the two evaluators.
The output is similar to this:
3

Analyze Experiment Results

Find the most frequent type of error in the results.Go to Datasets and Experiments and open the support query dataset. The experiment shows in the dataset.The ground_truth_evaluator score is 0.53. This means that 53% of the classifications match the ground truth.The output_evaluator also gives feedback in natural language for each output. Use this feedback to find the errors.To show only the incorrect classifications, use this filter:
Hover over the output_evaluator score to see the feedback. This is one example:
This is the same broad-versus-specific error that you fixed in Part 1. To show all rows with the broad_vs_specific error type, use this filter:
The filter shows 30 rows. broad_vs_specific is the most frequent error type.Write down this instruction to fix broad_vs_specific errors. You add it to the prompt in Part 3.

Summary

You tested your prompt on a full dataset.
You have a baseline accuracy of 53%. An LLM evaluator explained each failure, so you did not read every input and output pair. You found the most frequent error type and wrote an instruction to fix it.

Next Steps

In Compare Prompt Versions, you add the new instruction to the prompt and change the model parameters. Then you run experiments with the new versions and compare them with the baseline.