Run experiments and evals from the platform

Most experiment workflows follow the same pattern: run a single-turn prompt task over a dataset, then score the outputs with a judge or a deterministic check. Arize AX handles both ends natively — use Prompt Playground to run and save the task, and the Evaluator Hub to define and trigger the eval.

Run the experiment from the platform

1. Define the task in Prompt Playground

Load your prompt and dataset into the Prompt Playground, run it over every row, and save the outputs as an experiment.

Prompt Playground dataset selector dropdown with a golden dataset selected and template variables highlighted in the prompt — Select a dataset in Prompt Playground to map rows to prompt variables

For a step-by-step walkthrough, see Test prompts on datasets.

2. Add an LLM-as-a-judge from the Evaluator Hub

Create a judge evaluator in the Evaluator Hub and attach it to your experiment. The hub gives you versioning, reuse across projects, and alignment tracking. See LLM as a judge for guidance on writing the prompt template.

Arize AX Evaluators page with Evaluator Hub tab selected, showing a table of saved LLM-as-a-judge evaluators with scope, judge model, maintainer, and usage count — Evaluator Hub listing saved LLM judges ready to attach to any experiment

Create Evaluator modal on the Experiments page for a span-level hallucination judge with model selector, prompt template editor, optional test on dataset with example preview and variable mapping, and Ask Alyx — Create an LLM evaluator directly from the experiment page

3. Add a code evaluator (when rules are deterministic)

For objective checks — JSON validity, keyword presence, regex matching — create a code evaluator instead of an LLM judge. Code evals run instantly and have no inference cost.

4. Run, view, and compare

Choose the experiments you want to evaluate, click Run, and view the scored results inline. Use Compare experiments to diff outputs and eval scores across prompt or model versions side by side.

Run on Experiment modal on a dataset Experiments tab showing selected experiments, options to skip or override existing evaluation labels with the same eval column name, and Cancel and Run actions — Run evaluators on selected experiments from the Run on Experiment modal

Compare Experiments table tab showing dataset rows, experiment outputs side by side, Evals column with correct and incorrect tags, and an open popover with score, label, and explanation for an evaluator — Compare eval scores across experiment versions in the Compare Experiments table

When to drop down to code

Use the SDK directly when the platform cannot express your workflow:

Multi-step agent tasks — loops with tool use, branching logic, or state that must be driven from your code.
Tightly coupled deterministic logic — checks that depend on internal Python types or business logic that don’t map cleanly to a code-eval template.
CI/CD pipelines — when the experiment and score must run inside a Python test runner and gate a deployment. See GitHub Action basics and GitLab CI/CD basics.

Alternative: define evaluators as a class in the SDK

The class-based API is an alternative for the cases above, or if you prefer object-oriented patterns over functional ones.

Eval class inputs

Parameter name	Description	Example
`input`	experiment run input	`def evaluate(self, input, **kwargs)`
`output`	experiment run output	`def evaluate(self, output, **kwargs)`
`dataset_row`	entire row of the dataset as a dict	`def evaluate(self, dataset_row, **kwargs)`
`metadata`	experiment metadata	`def evaluate(self, metadata, **kwargs)`

from arize.experiments import Evaluator

class ExampleAll(Evaluator):
    def evaluate(self, input, output, dataset_row, metadata, **kwargs) -> EvaluationResult:
        pass

class ExampleOutput(Evaluator):
    def evaluate(self, output, **kwargs) -> EvaluationResult:
        pass

class ExampleDatasetRow(Evaluator):
    def evaluate(self, dataset_row, **kwargs) -> EvaluationResult:
        pass

EvaluationResult outputs

Return type	Description
`EvaluationResult`	Score, label, and explanation
`float`	Numeric score only
`string`	Label only

class ExampleResult(Evaluator):
    def evaluate(self, input, output, dataset_row, metadata, **kwargs) -> EvaluationResult:
        return EvaluationResult(score=score, label=label, explanation=explanation)

class ExampleScore(Evaluator):
    def evaluate(self, output, **kwargs) -> EvaluationResult:
        return 1.0

class ExampleLabel(Evaluator):
    def evaluate(self, output, **kwargs) -> EvaluationResult:
        return "good"

Code evaluator as a class

from arize.experiments import EvaluationResult, Evaluator

class MatchesExpected(Evaluator):
    annotator_kind = "CODE"
    name = "matches_expected"

    def evaluate(self, output, dataset_row, **kwargs) -> EvaluationResult:
        expected_output = dataset_row.get("expected")
        label = expected_output == output
        score = float(label)
        return EvaluationResult(score=score, label=label)

    async def async_evaluate(self, *, output, dataset_row, **kwargs) -> EvaluationResult:
        return self.evaluate(output=output, dataset_row=dataset_row)

Pass the evaluator class instance to evaluators when running the experiment:

experiment, results_df = client.experiments.run(
    name="my-experiment",
    dataset_id=dataset_id,
    task=my_task,
    evaluators=[MatchesExpected()],
)

For LLM-as-a-judge in code, define the judge in the Evaluator Hub and trigger it on your experiment via ax tasks trigger-run rather than calling llm_classify inside an Evaluator subclass.

Multiple evaluators on experiment runs

Pass multiple evaluators in a list. Arize creates an evaluation run for every combination of experiment run and evaluator.

experiment, results_df = client.experiments.run(
    name="multi-eval-experiment",
    dataset_id=dataset_id,
    task=task,
    evaluators=[
        ContainsKeyword("hello"),
        MatchesRegex(r"\d+"),
        custom_evaluator_function,
    ]
)

Evaluate experiment

Run evaluators on experiments from the UI overview.

Run experiment

Create datasets and run experiments end to end.

Run offline evals on experiments

Full guide to scoring experiments — UI, Arize Skills, and By Code tabs.

Create evaluators

Build LLM-as-a-judge and code evaluators in the Evaluator Hub.

Prompt Playground

Run prompts over datasets and save outputs as experiments.

Compare experiments

Diff outputs and eval scores across prompt or model versions.

Quickstart

Instrument

Observe

Evaluate

Improve

Agents

Machine Learning

Settings

Security

Run experiments and evals from the platform

Run the experiment from the platform

1. Define the task in Prompt Playground

2. Add an LLM-as-a-judge from the Evaluator Hub

3. Add a code evaluator (when rules are deterministic)

4. Run, view, and compare

When to drop down to code

Alternative: define evaluators as a class in the SDK

Eval class inputs

EvaluationResult outputs

Code evaluator as a class

Multiple evaluators on experiment runs

Evaluate experiment

Run experiment

Run offline evals on experiments

Create evaluators

Prompt Playground

Compare experiments

​Run the experiment from the platform

​1. Define the task in Prompt Playground

​2. Add an LLM-as-a-judge from the Evaluator Hub

​3. Add a code evaluator (when rules are deterministic)

​4. Run, view, and compare

​When to drop down to code

​Alternative: define evaluators as a class in the SDK

​Eval class inputs

​EvaluationResult outputs

​Code evaluator as a class

​Multiple evaluators on experiment runs

​Related

Evaluate experiment

Run experiment

Run offline evals on experiments

Create evaluators

Prompt Playground

Compare experiments

Run the experiment from the platform

1. Define the task in Prompt Playground

2. Add an LLM-as-a-judge from the Evaluator Hub

3. Add a code evaluator (when rules are deterministic)

4. Run, view, and compare

When to drop down to code

Alternative: define evaluators as a class in the SDK

Eval class inputs

EvaluationResult outputs

Code evaluator as a class

Multiple evaluators on experiment runs

Related