support-classifier prompt on a dataset of support queries. At the end, you have a baseline accuracy score and a new instruction that fixes the most frequent type of error.
You do these tasks:
- Upload a dataset of support queries.
- Run an experiment with the current prompt.
- Score each output and generate feedback with evaluators.
- Analyze the results to find the most frequent error type.
Follow along with code: This guide has a companion notebook with runnable code examples. Find it here, and go to Part 2: Test Prompts at Scale.
1
Load Dataset of Inputs
Upload a dataset of support queries. Each row has a query and its correct class.
- UI
- Python
- TypeScript
-
Download
support_queries.csvhere. - Go to Datasets and Experiments. Click Create Dataset and upload the file.
-
For Input keys, select
query. This is the input column. -
For Output keys, select
ground_truth. This is the correct output. -
Click Create Dataset. Open the new dataset.
Upload dataset to Phoenix
2
Run Experiment with Our Current Prompt
Run the current prompt on all the examples in the dataset. The result is your baseline accuracy.The output is similar to this:
Define Task Function
Define a task function. The task generates an output for each input in the dataset. Here, the task asks the LLM to classify a support query.- Python
- TypeScript
Define Evaluators
Define evaluators. An evaluator scores each output and can explain why the output is correct or incorrect.Use two evaluators:ground_truth_evaluator– Checks if the predicted class matches the ground truth.output_evaluator– Uses an LLM to analyze each classification. It returns these fields:explanation– Why the classification is correct or incorrect.confusion_reason– If incorrect, why the model might have made the wrong choice.error_type– If incorrect, what kind of error occurred (broad_vs_specific,keyword_bias,multi_intent_confusion,ambiguous_query,off_topic,paraphrase_gap, orother).evidence_span– The exact phrase in the query that supports the correct classification.prompt_fix_suggestion– An instruction to add to the classifier prompt to prevent this type of error.
analysis_evaluator_template. Copy the full template from the “Define Evaluators” section of the notebook.- Python
- TypeScript
Run Experiment
Run the experiment with the task and the two evaluators.- Python
- TypeScript
3
Analyze Experiment Results
Find the most frequent type of error in the results.Go to Datasets and Experiments and open the support query dataset. The experiment shows in the dataset.The Hover over the This is the same broad-versus-specific error that you fixed in Part 1. To show all rows with the The filter shows 30 rows.
ground_truth_evaluator score is 0.53. This means that 53% of the classifications match the ground truth.The output_evaluator also gives feedback in natural language for each output. Use this feedback to find the errors.To show only the incorrect classifications, use this filter:output_evaluator score to see the feedback. This is one example:broad_vs_specific error type, use this filter:broad_vs_specific is the most frequent error type.Write down this instruction to fix broad_vs_specific errors. You add it to the prompt in Part 3.Summary
You tested your prompt on a full dataset.You have a baseline accuracy of 53%. An LLM evaluator explained each failure, so you did not read every input and output pair. You found the most frequent error type and wrote an instruction to fix it.

