Evaluate a Prompt Change in Code

Evaluate scores two prompts with an LLM judge in one short script. This page is the longer code walkthrough: a deterministic code evaluator instead of a judge, examples with ids and metadata, and a dataset you built from traces in the UI. The examples are billing-support replies. It needs a running Phoenix and no model key. You define two tasks, a baseline and an edited prompt, score both with one deterministic evaluator, and read the difference. The code lives in examples/quickstarts, ready to run.

Build and run the experiment

The script creates its own dataset, so you can run it without traces in Phoenix. Choose Python or TypeScript in each step.

Install experiment packages

Python
TypeScript

pip install -U arize-phoenix-client

npm install @arizeai/phoenix-client tsx

Create an experiment file

The dataset uses two billing support examples; replace them with selected traces or examples from your agent when you are ready.

Python
TypeScript

Create a file named experiment.py.

from phoenix.client import Client

client = Client()

dataset = client.datasets.create_dataset(
    name="support-quickstart",
    dataset_description="Support reply examples from the Phoenix quickstart.",
    examples=[
        {
            "id": "invoice-change",
            "input": {"text": "Draft a short reply about why an invoice changed."},
            "output": {
                "text": "Explain that invoice totals can change when usage, taxes, or plan adjustments are applied."
            },
            "metadata": {"kind": "billing"},
        },
        {
            "id": "renewal-plan-change",
            "input": {
                "text": "A customer says their renewal invoice is higher after changing plans."
            },
            "output": {
                "text": "Explain that plan changes can cause prorated usage charges and updated taxes."
            },
            "metadata": {"kind": "billing"},
        },
    ],
)


def baseline_task(input):
    if "renewal" in input["text"]:
        return "Renewal amounts can change after account updates."
    return "Invoice totals can change when billing settings change."


def edited_task(input):
    if "renewal" in input["text"]:
        return (
            "Plan changes can create prorated usage charges and updated taxes on a renewal invoice."
        )
    return "Invoice totals can change when usage, taxes, or plan adjustments are applied."


def covers_billing_context(output):
    text = output.lower()
    has_usage = "usage" in text or "prorat" in text
    has_taxes = "tax" in text
    has_plan = "plan" in text
    return has_usage and has_taxes and has_plan


client.experiments.run_experiment(
    dataset=dataset,
    task=baseline_task,
    evaluators=[covers_billing_context],
    experiment_name="support-response-baseline",
)

client.experiments.run_experiment(
    dataset=dataset,
    task=edited_task,
    evaluators=[covers_billing_context],
    experiment_name="support-response-edited",
)

Create a file named experiment.ts.

import { createClient } from "@arizeai/phoenix-client";
import { createDataset } from "@arizeai/phoenix-client/datasets";
import {
  asExperimentEvaluator,
  runExperiment,
} from "@arizeai/phoenix-client/experiments";
import type { Example } from "@arizeai/phoenix-client/types/datasets";

const client = createClient();

const dataset = await createDataset({
  client,
  name: "support-quickstart",
  description: "Support reply examples from the Phoenix quickstart.",
  examples: [
    {
      id: "invoice-change",
      input: { text: "Draft a short reply about why an invoice changed." },
      output: {
        text: "Explain that invoice totals can change when usage, taxes, or plan adjustments are applied.",
      },
      metadata: { kind: "billing" },
    },
    {
      id: "renewal-plan-change",
      input: {
        text: "A customer says their renewal invoice is higher after changing plans.",
      },
      output: {
        text: "Explain that plan changes can cause prorated usage charges and updated taxes.",
      },
      metadata: { kind: "billing" },
    },
  ],
});

const baselineTask = (example: Example): string => {
  const input = example.input as { text: string };

  if (input.text.includes("renewal")) {
    return "Renewal amounts can change after account updates.";
  }
  return "Invoice totals can change when billing settings change.";
};

const editedTask = (example: Example): string => {
  const input = example.input as { text: string };

  if (input.text.includes("renewal")) {
    return "Plan changes can create prorated usage charges and updated taxes on a renewal invoice.";
  }
  return "Invoice totals can change when usage, taxes, or plan adjustments are applied.";
};

const coversBillingContext = asExperimentEvaluator({
  name: "covers_billing_context",
  kind: "CODE",
  evaluate: ({ output }) => {
    const text = String(output).toLowerCase();
    const hasUsage = text.includes("usage") || text.includes("prorat");
    const hasTaxes = text.includes("tax");
    const hasPlan = text.includes("plan");
    const passes = hasUsage && hasTaxes && hasPlan;
    return {
      label: passes ? "True" : "False",
      score: passes ? 1 : 0,
    };
  },
});

await runExperiment({
  client,
  dataset,
  task: baselineTask,
  evaluators: [coversBillingContext],
  experimentName: "support-response-baseline",
});

await runExperiment({
  client,
  dataset,
  task: editedTask,
  evaluators: [coversBillingContext],
  experimentName: "support-response-edited",
});

Each task takes one dataset example and returns an agent output. Swap in your agent’s entry point. The evaluator is intentionally deterministic: it checks for the billing context the response must include before you add broader LLM-as-a-judge evals.

Run the experiment

Python
TypeScript

python experiment.py

If you already created a dataset from traces in the UI, replace the create_dataset call with:

dataset = client.datasets.get_dataset(dataset="<YOUR DATASET NAME>")

npx tsx experiment.ts

If you already created a dataset from traces in the UI, replace the createDataset call with:

import { getDataset } from "@arizeai/phoenix-client/datasets";

const dataset = await getDataset({ client, dataset: { datasetName: "<YOUR DATASET NAME>" } });

Review the scores

Head back to Phoenix and open the dataset’s Experiments tab. You can compare outputs, evaluator labels, scores, and explanations for each example.

Phoenix experiment comparison view showing a baseline support response scoring 0 and an edited response scoring 1 — Side-by-side experiment comparison

Mark a persistent baseline

The comparison above is transient. To pin one experiment as the reference point that later comparisons measure against, see Set a Baseline Experiment.

Tracing

Evaluation

Datasets & Experiments

Prompts

Settings

Concepts

Resources

Evaluate a Prompt Change in Code

Build and run the experiment

Mark a persistent baseline

​Build and run the experiment

​Mark a persistent baseline

Build and run the experiment

Mark a persistent baseline