> ## Documentation Index
> Fetch the complete documentation index at: https://arizeai-433a7140.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Experiments with Code Evals

> Learn how to define and run experiments to systematically evaluate your AI application using code-based evaluators and ground truth data.

<div style={{ display: 'flex', gap: '1rem', marginBottom: '1rem' }}>
  <Card title="Follow with Complete Python Notebook" icon="python" href="https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/experiments/python_experiments_quickstart.ipynb" description="End-to-end Python notebook" />

  <Card title="Follow with Complete TypeScript Tutorial" icon="js" href="https://github.com/Arize-ai/phoenix/tree/main/js/examples/apps/experiments-quickstart" description="End-to-end TypeScript tutorial" />
</div>

In this section, you'll run a repeatable experiment that scores your agent's tool outputs against ground truth reference using code-based evaluators. These evaluators are fast and typically perform deterministic checks for correctness, such as exact matches or comparisons against expected labels.

## Parts of an Experiment

An experiment consists of four main components:

<Steps>
  <Step title={<span className="step-title">Task Function</span>}>
    The task function represents the work your application performs for a single dataset example. You can pass the input (or other fields) from your dataset into this function, which produces an output for evaluation.

    The task can be as simple or complex as your real system—it might call a single LLM, invoke a multi-step pipeline, retrieve context, or execute tool calls. What matters is that the task mirrors how your application behaves in practice so that experiment results reflect real-world performance.

    By defining the task once and running it across all dataset examples, you ensure that every version of your system is evaluated under consistent conditions.
  </Step>

  <Step title={<span className="step-title">Evaluators</span>}>
    Evaluators determine how Phoenix measures the quality of each task output. They take the task output and optionally compare it against a reference or expected output from the dataset.

    **Code-based evaluators** are deterministic Python functions useful when you have a clear, programmatic definition of correctness, such as exact match or comparing outputs against ground truth labels.

    Alternatively, [**LLM as a Judge evaluators**](/docs/phoenix/datasets-and-experiments/tutorial/run-experiments-with-llm-judge) use an LLM to assess output quality and are useful for subjective quality assessments or when you don't have ground truth.

    You can use one or multiple evaluators in the same experiment to capture different dimensions of quality.
  </Step>

  <Step title={<span className="step-title">Dataset Selection</span>}>
    Next, you connect the experiment to your dataset. You can run an experiment over the entire dataset or select a specific subset. This configuration step defines the exact scope and rigor of your experiment.
  </Step>

  <Step title={<span className="step-title">Run the Experiment</span>}>
    Once the task, evaluators, dataset, and configuration are defined, you can run the experiment. Phoenix executes the task across all selected examples, applies evaluators to each output, and stores the results.

    Each experiment run is tracked with its configuration, outputs, and evaluation scores, making it easy to compare experiments over time. This allows you to answer questions like whether a prompt change improved accuracy, whether a model swap introduced regressions, or how performance differs across different datasets.
  </Step>
</Steps>

## Why Use Experiments Instead of Regular Evaluations?

While you can run individual evaluations on traces or ad-hoc examples, experiments provide a structured, repeatable framework for systematic evaluation. Experiments offer:

* **Consistency**: Experiments ensure every version of your system is evaluated under identical conditions, using the same dataset and task. This eliminates variability from manual testing or one-off evaluations.

* **Comparability**: Experiments track configuration, outputs, and scores over time, making it easy to compare different versions of your agent. You can answer questions like "Did my prompt change improve accuracy?" or "Did switching models introduce regressions?"

* **Systematic Coverage**: Experiments run your task function across all dataset examples, ensuring comprehensive evaluation rather than spot-checking a few cases.

## Code-Based Evaluator for Tool Call Accuracy

This experiment evaluates the accuracy of the agent's `classify_ticket` tool by comparing its output against ground truth labels in the dataset. Since we have a golden dataset with ground truth available, we can use a code-based evaluator for fast, deterministic evaluation.

### Define the Task Function

In this example, the task function takes an example from the dataset, extracts the user’s query, and passes it to the `classify_ticket` tool. The tool invokes the underlying `classify_ticket` function, which makes an LLM call to determine the ticket’s category—billing, technical, account, or other. The task function then returns this classification as its final output.

This simulates how your agent performs a tool call, allowing us to isolate and evaluate a specific capability. For the full implementation, see the reference tutorials linked above.

This task function will be used for all dataset examples.

<CodeGroup>
  ```python Python theme={null}
  def classify_ticket_task(input):
      """
      Task used specifically for evaluating tool call accuracy.
      """
      query = input.get("query")
      classification = classify_ticket(query)
      return classification
  ```

  ```typescript TypeScript theme={null}
  import type { Example } from "@arizeai/phoenix-client/types/datasets";

  async function classifyTicketTask(example: Example) {
    return classifyTicket(example.input.query as string);
  }
  ```
</CodeGroup>

### Define the Code-Based Evaluator

Since our dataset has ground truth available in the `expected_category` field, we can use a code-based evaluator to check if the task output matches what we expect. This evaluator compares the tool's output against the reference output from the dataset:

<CodeGroup>
  ```python Python theme={null}
  # Define Code-Based Evaluator for Tool Call Accuracy
  from phoenix.client.experiments import create_evaluator

  @create_evaluator(kind="CODE", name="tool-call-accuracy")
  def tool_call_accuracy(output: str, expected: dict) -> bool:
      """
      Code-based evaluator that checks if the classify_ticket tool output
      matches the expected category from the dataset.
      """
      if expected is None:
          return None
      expected_category = expected.get("expected_category")
      return output.strip().lower() == expected_category.strip().lower()
  ```

  ```typescript TypeScript theme={null}
  import { asExperimentEvaluator } from "@arizeai/phoenix-client/experiments";
  import type { EvaluatorParams } from "@arizeai/phoenix-client/types/experiments";

  // Define Code-Based Evaluator for Tool Call Accuracy
  const toolCallAccuracyEvaluator = asExperimentEvaluator({
    name: "tool-call-accuracy",
    kind: "CODE",
    evaluate: ({
      output,
      expected,
    }: EvaluatorParams): { score: number; label: string } => {
      if (!expected) {
        return { score: 0, label: "missing_expected" };
      }

      const expectedCategory = (
        expected as { expected_category: string }
      ).expected_category.trim().toLowerCase();
      const actualOutput = String(output).trim().toLowerCase();

      const isCorrect = actualOutput === expectedCategory;
      return {
        score: isCorrect ? 1 : 0,
        label: isCorrect ? "correct" : "incorrect",
      };
    },
  });
  ```
</CodeGroup>

### Run the Experiment

Now we can run the experiment on our dataset with ground truth. We'll use the golden dataset we created earlier:

<CodeGroup>
  ```python Python theme={null}
  from phoenix.client.experiments import run_experiment

  golden_dataset = px_client.datasets.get_dataset(
      dataset="support-ticket-queries"
  )

  experiment = run_experiment(
      dataset=golden_dataset,
      task=classify_ticket_task,
      evaluators=[tool_call_accuracy],
      experiment_name="tool call accuracy experiment",
      experiment_description="Evaluating classify_ticket tool accuracy against ground truth labels using a code-based evaluator"
  )
  ```

  ```typescript TypeScript theme={null}
  import { runExperiment } from "@arizeai/phoenix-client/experiments";

  const experiment = await runExperiment({
    dataset: { datasetName: "support-ticket-queries" },
    experimentName: "tool call accuracy experiment",
    experimentDescription:
      "Evaluating classify_ticket tool accuracy against ground truth labels using a code-based evaluator",
    task: classifyTicketTask,
    evaluators: [toolCallAccuracyEvaluator],
  });
  ```
</CodeGroup>

In the Phoenix UI, you can click into the experiment to inspect the results:

* **Task function traces** let you drill into any run to see the exact inputs, tool calls, and outputs—useful for debugging when an example fails or understanding why the model behaved a certain way.

* **Scores per example** show which inputs your system got right or wrong, so you can spot failure patterns and prioritize fixes.

* **Aggregate experiment performance metrics** give you a single view of how this run did overall, making it easy to compare experiments and track whether changes improve or regress quality.

<Frame caption="Running an experiment with a code-based evaluator">
  <video src="https://storage.googleapis.com/arize-phoenix-assets/assets/images/python_experiment_quickstart_code_eval.mp4" width="100%" height="100%" style={{ display: 'block', objectFit: 'fill', backgroundColor: 'transparent' }} controls autoPlay muted loop />
</Frame>

# Next Steps

Now that you know how to run experiments with ground truth, you can also evaluate your agent using LLM as a Judge evaluators for more subjective quality assessments.

<Card title="Run Experiments with LLM as a Judge" icon="brain" href="/docs/phoenix/datasets-and-experiments/tutorial/run-experiments-with-llm-judge" description="LLM-as-judge experiments" />