Skip to main content
Your online eval found failures and you have a fix in mind. Test it first with an offline eval, which scores prompts against a fixed dataset of test cases. This guide uses a three-row dataset about a fictional airline.
1

Create a dataset

  1. Download travel-support.csv: three Basic fare questions, each with its correct answer.
  2. Go to Datasets → New Dataset → Upload CSV, name it travel-support, add the file, and click Create Dataset.
The answers differ from industry norms, so the prompt has to supply the policy. Already have failing traces? Build a dataset from them instead.
2

Run a baseline

  1. Go to Playgrounds and choose travel-support from Select a Dataset. Its columns become variables like {question}.
  2. Use this as the user prompt:

Selecting a dataset in the Prompt Playground.

3

Add an evaluator

  1. Click Add Evaluator → New Evaluator and pick the Q&A template, which labels each answer correct or incorrect against reference data.
  2. Under Map Variable to Data:
  3. Save the evaluator and click Run.
Expect incorrect labels, since the prompt doesn’t know the policy. Click View Experiment and hover a label to read why.
4

Test the fix

  1. Click Add → Duplicate and replace the copy’s user prompt with:
  2. Click Run All. Both prompts use the same dataset and evaluator, so the only difference is your change.
  3. To compare in detail, open the dataset’s Experiments tab, select both runs, and click Compare.
Compare Experiments view showing two experiments side by side

Two experiments compared side by side on the same evaluator.

If the new prompt wins, click Save Prompt to version it in Prompt Hub.
To find failures you didn’t know to look for, use Signal.

​
Other ways to run experiments

With your coding agent, using Arize Skills (arize-dataset, arize-experiment, arize-evaluator, and arize-prompt-optimization). See coding agent setup for authentication.
With Alyx:
In code, from the Python SDK, TypeScript SDK, or ax CLI:

Previous: Evaluate production traffic

Next: Signal to find issues