> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Run evals on experiments

> Build a small dataset, run a prompt over it as an experiment, score it with an LLM judge, and prove a prompt change helped before you ship it.

Your [online eval](/docs/ax/evaluate/run-evals-on-traces) found failures and you have a fix in mind. Test it first with an [**offline eval**](/docs/ax/evaluate/run-evals-on-experiments), which scores prompts against a fixed [dataset](/docs/ax/develop/datasets-and-experiments) of test cases. This guide uses a three-row dataset about a fictional airline.

<Steps>
  <Step title="Create a dataset">
    1. Download [travel-support.csv](https://storage.googleapis.com/arize-phoenix-assets/assets/images/arize-docs-images/evaluate/travel-support.csv): three Basic fare questions, each with its correct answer.
    2. Go to **Datasets** → **New Dataset** → **Upload CSV**, name it `travel-support`, add the file, and click **Create Dataset**.

    The answers differ from industry norms, so the prompt has to supply the policy. Already have failing traces? [Build a dataset from them](/docs/ax/improve/build-a-dataset) instead.
  </Step>

  <Step title="Run a baseline">
    1. Go to **Playgrounds** and choose `travel-support` from **Select a Dataset**. Its columns become variables like `{question}`.
    2. Use this as the user prompt:

       ```text theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
       You are a travel support agent. Answer the customer's question.

       Question: {question}
       ```

    <Frame caption="Selecting a dataset in the Prompt Playground.">
      <video
        src="https://storage.googleapis.com/arize-phoenix-assets/assets/images/arize-docs-images/load-dataset-playground-arize.mp4"
        width="100%"
        height="100%"
        style={{
          display: 'block',
          objectFit: 'fill',
          backgroundColor: 'transparent',
        }}
        controls
        autoPlay
        muted
        loop
      />
    </Frame>
  </Step>

  <Step title="Add an evaluator">
    1. Click **Add Evaluator** → **New Evaluator** and pick the **Q\&A** template, which labels each answer `correct` or `incorrect` against reference data.

    2. Under **Map Variable to Data**:

       | Template variable | Map to |
       | - | - |
       | `question` | the `question` column |
       | `input` | the `expected_answer` column |
       | `output` | the experiment's output |

    3. Save the evaluator and click **Run**.

    Expect `incorrect` labels, since the prompt doesn't know the policy. Click **View Experiment** and hover a label to read why.
  </Step>

  <Step title="Test the fix">
    1. Click **Add** → **Duplicate** and replace the copy's user prompt with:

       ```text theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
       You are a travel support agent. Answer the customer's question using this policy.

       Basic fare policy:
       - Full refund if cancelled within 48 hours of booking.
       - Can be changed once for a 40 dollar fee, plus any price difference.
       - Includes one checked bag. A second bag is 45 dollars.

       Question: {question}
       ```

    2. Click **Run All**. Both prompts use the same dataset and evaluator, so the only difference is your change.

    3. To compare in detail, open the dataset's **Experiments** tab, select both runs, and click [**Compare**](/docs/ax/develop/datasets-and-experiments/compare-experiments).

    <Frame caption="Two experiments compared side by side on the same evaluator.">
      <img src="https://storage.googleapis.com/arize-phoenix-assets/assets/images/arize-docs-images/get-started-images/23-experiment_comparison.png" alt="Compare Experiments view showing two experiments side by side" />
    </Frame>

    If the new prompt wins, click **Save Prompt** to version it in [Prompt Hub](/docs/ax/prompts/prompt-hub).
  </Step>
</Steps>

To find failures you didn't know to look for, use [Signal](/docs/ax/get-started/get-started-managed-agent).

## Other ways to run experiments

**With your coding agent**, using [Arize Skills](/docs/ax/skills/overview) ([`arize-dataset`](https://github.com/Arize-ai/arize-skills/blob/main/skills/arize-dataset/SKILL.md), [`arize-experiment`](https://github.com/Arize-ai/arize-skills/blob/main/skills/arize-experiment/SKILL.md), [`arize-evaluator`](https://github.com/Arize-ai/arize-skills/blob/main/skills/arize-evaluator/SKILL.md), and [`arize-prompt-optimization`](https://github.com/Arize-ai/arize-skills/blob/main/skills/arize-prompt-optimization/SKILL.md)). See [coding agent setup](/docs/ax/set-up-with-ai-assistants) for authentication.

```text theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
Download https://storage.googleapis.com/arize-phoenix-assets/assets/images/arize-docs-images/evaluate/travel-support.csv and create an Arize AX dataset called "travel-support" from it. Add a Q&A evaluator that checks each answer against the expected_answer column. Run an experiment over the dataset with a travel support prompt, and tell me which examples scored lowest and why.
```

**With [Alyx](/docs/ax/alyx)**:

```text theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
Run my prompt over the travel-support dataset as an experiment, score each answer against expected_answer with the Q&A evaluator, then add the Basic fare policy to the prompt, rerun, and compare the two experiments.
```

**In code**, from the [Python SDK](/docs/api-clients/python/overview), [TypeScript SDK](/docs/api-clients/typescript/version-1/overview), or [`ax` CLI](/docs/api-clients/cli/overview):

| Step | Python SDK | TypeScript SDK | CLI |
| - | - | - | - |
| Filter spans by eval result | [Link](/docs/api-clients/python/version-8/client-resources/spans) | [Link](/docs/api-clients/typescript/version-1/client-resources/spans) | [Link](/docs/api-clients/cli/spans) |
| Create a dataset from traces | [Link](/docs/api-clients/python/version-8/client-resources/datasets) | [Link](/docs/api-clients/typescript/version-1/client-resources/datasets) | [Link](/docs/api-clients/cli/datasets) |
| Manage prompts | [Link](/docs/api-clients/python/version-8/client-resources/prompts) | [Link](/docs/api-clients/typescript/version-1/client-resources/prompts) | [Link](/docs/api-clients/cli/prompts) |
| Run experiments | [Link](/docs/api-clients/python/version-8/client-resources/experiments) | [Link](/docs/api-clients/typescript/version-1/client-resources/experiments) | [Link](/docs/api-clients/cli/experiments) |

<CardGroup cols={2}>
  <Card title="Previous: Evaluate production traffic" icon="arrow-left" href="/docs/ax/get-started/get-started-evaluations" />

  <Card title="Next: Signal to find issues" icon="arrow-right" href="/docs/ax/get-started/get-started-managed-agent" />
</CardGroup>