> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent playground

> Launch a dataset against a registered remote agent, override config per run, and compare runs in the agent playground.

Once you've [registered an agent](/docs/ax/improve/setup-agent-endpoint) and [wired up tracing](/docs/ax/improve/agent-tracing-context), running an experiment against it is a no-code workflow. This page covers the agent playground UI for kicking off, monitoring, and comparing runs.

## Prerequisites

* A dataset in your space ([Build a dataset](/docs/ax/improve/build-a-dataset)).
* A registered agent configuration ([Setting up your remote agent](/docs/ax/improve/setup-agent-endpoint)).
* *Optional but recommended:* tracing wired into your agent ([Setting up tracing](/docs/ax/improve/agent-tracing-context)).

## Launch an experiment

<Frame>
  <video
    src="https://storage.googleapis.com/arize-assets/doc-images/agent%20experiments/agent-experiment-run-experiment.mp4"
    alt="Launching an agent experiment against a dataset in the Agent Playground"
    width="100%"
    height="100%"
    style={{
  display: 'block',
  objectFit: 'fill',
  backgroundColor: 'transparent',
}}
    controls
    autoPlay
    muted
    loop
  />
</Frame>

<Steps>
  <Step title="Open the dataset">
    Navigate to **Datasets** in the left nav and click the dataset you want to run against.
  </Step>

  <Step title="Click New Experiment → Run in Agent Playground">
    The agent playground modal opens with the dataset already selected.
  </Step>

  <Step title="Pick an agent">
    Use the **Agent** dropdown to select one of your registered agent configurations. The endpoint URL and auth from the configuration are used automatically — you don't re-enter them per run.
  </Step>

  <Step title="Choose a preset (or write custom config)">
    If the agent has [request presets](/docs/ax/improve/setup-agent-endpoint#optional-add-request-presets), pick one from the **Preset** dropdown. The preset's fields are merged into the request editor at the top level, so a preset that sets `config` replaces `config` but leaves your `goal` placeholder untouched.

    You can edit the JSON before running — useful when you want to tweak one parameter from a known-good baseline without saving a new preset. Once you edit, **Save as preset** becomes available.
  </Step>

  <Step title="Edit the request body">
    The **Request** editor holds the JSON Arize sends to your endpoint. It's pre-filled from the agent's input schema. Use `{{dataset.column_name}}` to interpolate from each dataset row (the editor autocompletes column names):

    ```json theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
    {
      "goal": "{{dataset.input}}",
      "config": { "model": "claude-sonnet-4-6", "max_turns": 12 }
    }
    ```

    These fields are sent at the top level of the request body, with `arize_metadata` added beside them. The panel next to the editor previews the metadata Arize will inject.
  </Step>

  <Step title="(Optional) Limit to a subset">
    For quick sanity checks, run on a subset of the dataset before committing to the full run. Useful for validating the request shape with one or two rows first.
  </Step>

  <Step title="(Optional) Add evaluators">
    Attach evaluators to score each run's output automatically. See [Evals overview](/docs/ax/evaluate/run-evals-on-experiments) for setup.
  </Step>

  <Step title="Click Run">
    Arize fans out the dataset against your endpoint in parallel, applies retries on transient failures, and streams results into a new experiment.
  </Step>
</Steps>

## Watch a run in progress

As the experiment runs, the experiments view streams rows in real time:

* **Status column** — pending, running, succeeded, failed.
* **Output column** — the agent's response body.
* **Latency / tokens** — populated from the spans your agent emitted (if traced).
* **Evaluator scores** — computed as each row completes.

If a row fails (timeout, HTTP error, agent exception), the error message appears in the run's detail panel along with the request body that was sent — useful for debugging and re-running just the failed subset.

## Inspecting an individual run

Click any row to see:

* **Input** — exact JSON body sent to your endpoint: your hydrated fields plus the `arize_metadata` Arize appended.
* **Output** — full response body returned.
* **Trace** — if your agent is traced, the linked trace tree (CHAIN, LLM, TOOL spans). Traces live in the space's **Agent Experiment Traces** project, not your app's tracing project.
* **Headers** — the request headers Arize sent, including `traceparent`, `Authorization`, and any custom headers you configured.
* **Evaluator scores** — per-evaluator pass/fail and reasoning.

The trace link is the highest-leverage debugging tool: when an agent run produces an unexpected output, opening the trace shows you which tool was called, what the LLM was prompted with at each turn, and where the chain diverged.

## Comparing runs

After you have two or more experiments on the same dataset, comparison is the point.

### From the experiments tab

Open **Datasets → your dataset → Experiments**, multi-select the runs you want to compare, and click **Compare**. The comparison view shows:

* **Side-by-side outputs** for each dataset row across the selected runs.
* **Evaluator deltas** — which rows improved, regressed, or stayed flat.
* **Summary metrics** — pass rate, average latency, token counts per run.
* **Tool-call patterns** — if traced, you can see which runs called different tools or took different paths.

### Common comparison patterns

| Question                                       | How to set it up                                                                |
| ---------------------------------------------- | ------------------------------------------------------------------------------- |
| Did the new model help?                        | Run two experiments, same dataset and preset, vary only `config.model`.         |
| Did the prompt change break anything?          | Run baseline before the deploy, then run again after. Compare evaluator deltas. |
| Which preset is best for prod?                 | Run each preset against the same dataset. Eyeball the summary table.            |
| Is this regression specific to one input type? | Filter the comparison view by a metadata column (e.g. `category`).              |

## Re-running failed rows

When a run finishes with failures, the **Run** button in the agent playground becomes **Retry**. Arize calls your endpoint again for the failed rows and merges the new results into the existing experiment — so you don't have to lose the successful rows when chasing one flake.

## Rate limits, timeouts, and retries

These are set on the [remote agent configuration](/docs/ax/improve/setup-agent-endpoint#optional-set-runtime-settings), not per launch:

* **Rate limit (requests/minute)** — caps how fast Arize calls your endpoint. Leave blank for the system default. Arize also backs off automatically when your endpoint returns `429`.
* **Request timeout** — per-request, in seconds. Default 120s, maximum 300s. Raise it for agents with long loops (e.g. multi-step research agents).
* **Retries** — transient failures (5xx, network errors) are retried automatically. This isn't user-configurable.

## Running from code or CLI

The agent playground is the UI path. To drive it from code (e.g. in CI), create a `RUN_EXPERIMENT` task with an `AGENT_CALL` run configuration. The run config takes the remote agent's `integration_id` and an `input_template` (the same JSON you'd put in the request editor; `{{column}}` and `{{dataset.column}}` are equivalent).

* **REST API** — `POST /v2/tasks` with `run_configuration.experiment_type: "AGENT_CALL"`. See the [REST API reference](/docs/ax/rest-reference/overview).
* **Python SDK** — `AgentCallRunConfig` from `arize.tasks`, passed as `run_configuration` when creating a task. See [Tasks (Python SDK)](/docs/api-clients/python/version-8/client-resources/tasks).
* **`ax` CLI** — register agents with `ax integrations create agent`, then launch with `ax tasks create-run-experiment --run-configuration @run_config.json`. See [`ax tasks`](/docs/api-clients/cli/tasks).

All paths produce the same experiment artifacts as the UI path, so you can mix and match (kick off via CI, debug in the UI).

## End-to-end example

Walking through the travel-agent demo (registered agent, dataset of 20 travel goals, three presets):

<Steps>
  <Step title="Pick the dataset">
    Open `travel-goals-v1` (20 goals like *"Plan a 3-day trip to Tokyo from SF in October"*, *"Weekend in NYC from Chicago"*).
  </Step>

  <Step title="Launch with the baseline preset">
    **New Experiment → Run in Agent Playground → travel-agent → Production baseline (Sonnet 4.5) → Run**. Wait \~3 minutes for 20 rows to complete.
  </Step>

  <Step title="Launch a second experiment with Opus">
    Same dataset, same agent, **Opus 4.7** preset. Run. \~5 minutes (Opus is slower).
  </Step>

  <Step title="Compare">
    **Compare Experiments** → see per-row output diffs. Opus produced richer itineraries on 16/20 rows, but average latency was 2.3× higher. Pass rate on the "produces a coherent multi-day plan" evaluator: Sonnet 18/20, Opus 20/20.
  </Step>

  <Step title="Inspect a regression">
    On row 7 (Lisbon), Sonnet picked a \$\$\$\$ hotel; Opus picked a \$\$ one. Open the Sonnet trace, see the `search_hotels` TOOL span — it ranked by rating, not by `max_price` constraint. Fix is a system prompt tweak.
  </Step>

  <Step title="Iterate">
    Update the agent's system prompt, redeploy, re-run the Sonnet experiment. Compare new Sonnet run to the previous one to confirm row 7 is fixed without breaking anything else.
  </Step>
</Steps>

That loop — run, compare, drill into trace, fix, re-run — is what agent experiments are designed to enable.

## Next

<CardGroup cols={2}>
  <Card title="Compare experiments" href="/docs/ax/improve/prompt-playground#compare-experiments">
    Side-by-side diffs and evaluator deltas across runs.
  </Card>

  <Card title="Run evals on experiments" href="/docs/ax/evaluate/run-evals-on-experiments">
    Add evaluators to score agent outputs.
  </Card>

  <Card title="CI/CD with experiments" href="/docs/ax/improve/ci-cd-for-automated-experiments">
    Trigger agent experiments from your deploy pipeline.
  </Card>

  <Card title="Code experiments" href="/docs/ax/improve/code-experiments">
    Drive agent experiments from Python / TypeScript / CLI.
  </Card>
</CardGroup>