Prompt Experimentation For Summarization Task

Summarization is a good task for prompt experimentation because the output is hard to eyeball at scale: two summaries can both look fine and still score very differently against a human reference. This guide builds a summarization task, scores it with ROUGE, and then runs the same dataset and evaluators against three different prompts so you can see which one actually wins in Arize AX.

Before you start

You need an Arize AX account and an OpenAI API key.

pip install "arize[Datasets]<8" openai datasets pyarrow rouge

export ARIZE_SPACE_ID="<your-space-id>"
export ARIZE_API_KEY="<your-api-key>"
export ARIZE_DEVELOPER_KEY="<your-developer-key>"
export OPENAI_API_KEY="<your-openai-api-key>"

Create a dataset

Connect to Arize AX with the Arize SDK and upload a dataset. We use ten rows from cnn_dailymail, which pairs each news article with a human-written summary. Those reference summaries are what the evaluators score against, so the dataset needs both an article and a summary column.

# Note: This example uses Python SDK v7
import os
from uuid import uuid1

from datasets import load_dataset
from arize.experimental.datasets import ArizeDatasetsClient
from arize.experimental.datasets.utils.constants import GENERATIVE

SPACE_ID = os.environ["ARIZE_SPACE_ID"]

hf_ds = load_dataset("abisee/cnn_dailymail", "3.0.0")
df = (
    hf_ds["test"]
    .to_pandas()
    .sample(n=10, random_state=0)
    .rename(columns={"highlights": "summary"})[["article", "summary"]]
    .reset_index(drop=True)
)

arize_client = ArizeDatasetsClient(
    developer_key=os.environ.get("ARIZE_DEVELOPER_KEY"),
    api_key=os.environ["ARIZE_API_KEY"],
)

dataset_name = "summarization-" + str(uuid1())[:5]
dataset_id = arize_client.create_dataset(
    space_id=SPACE_ID,
    dataset_name=dataset_name,
    dataset_type=GENERATIVE,
    data=df,
)
print(f"created dataset '{dataset_name}' (id: {dataset_id})")

Define our experiment

Now let’s define our experiment. The task summarizes one article per dataset row. Taking the prompt template and the model as arguments is what lets us reuse the identical task across prompt variants later — only the template changes.

from typing import Any, Dict

from openai import OpenAI

client = OpenAI()

MODEL = "gpt-5.5"


def summarize_article(dataset_row: Dict[str, Any], prompt_template: str, model: str) -> str:
    # dataset_row is a dictionary which contains every field for each row in your dataset
    article_text = dataset_row.get("article")
    response = client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "user",
                "content": prompt_template.format(article=article_text),
            },
        ],
    )
    return response.choices[0].message.content

Define our evaluators

Next, we can run evaluations on the results. Evaluators take the output of a task (in this case, a string) and grade it, often with the help of an LLM. In this case, we will create ROUGE evaluators to compare the LLM-generated summaries with the human reference summaries you uploaded as part of your dataset. There are several variants of ROUGE, but we’ll use ROUGE-1 F1 score for simplicity:

ROUGE-1 precision is the proportion of overlapping tokens (present in both reference and generated summaries) that are present in the generated summary (number of overlapping tokens / number of tokens in the generated summary)
ROUGE-1 recall is the proportion of overlapping tokens that are present in the reference summary (number of overlapping tokens / number of tokens in the reference summary)
ROUGE-1 F1 score is the harmonic mean of precision and recall, providing a single number that balances these two scores.

Higher ROUGE scores mean that a generated summary is more similar to the corresponding reference summary. Scores near 1 / 2 are considered excellent, and a model fine-tuned on this particular dataset achieved a rouge score of ~0.44. Logging precision and recall alongside F1 costs nothing extra and shows you why a prompt scored the way it did: a terse prompt tends to raise precision and lower recall, a verbose one does the reverse. Each evaluator returns an EvaluationResult. Return the full object rather than a bare float: label and explanation are reserved columns that cannot be null, so a float-only return fails the upload.

from arize.experimental.datasets.experiments.types import EvaluationResult
from rouge import Rouge


def _rouge_1(dataset_row, output) -> Dict[str, float]:
    # you can access the output of your task using the keyword output
    # you can also access any attribute of your dataset_row here
    expected = dataset_row.get("summary")
    return Rouge().get_scores(output, expected)[0]["rouge-1"]


def _result(score: float, metric: str) -> EvaluationResult:
    return EvaluationResult(
        score=score,
        label=f"{score:.2f}",
        explanation=f"ROUGE-1 {metric} against the reference summary",
    )


def rouge_1_f1_score(dataset_row, output) -> EvaluationResult:
    return _result(_rouge_1(dataset_row, output)["f"], "F1")


def rouge_1_precision(dataset_row, output) -> EvaluationResult:
    return _result(_rouge_1(dataset_row, output)["p"], "precision")


def rouge_1_recall(dataset_row, output) -> EvaluationResult:
    return _result(_rouge_1(dataset_row, output)["r"], "recall")


EVALUATORS = [rouge_1_f1_score, rouge_1_precision, rouge_1_recall]

Before spending a full experiment run, check the task and one evaluator on a single row:

from functools import partial

BASE_TEMPLATE = """
Summarize the article in two to four sentences:

ARTICLE
=======
{article}

SUMMARY
=======
"""

test_row = df.iloc[0].to_dict()
test_output = summarize_article(test_row, prompt_template=BASE_TEMPLATE, model=MODEL)
print("Generated:", test_output)
print("Reference:", test_row["summary"])
print("ROUGE-1 F1:", rouge_1_f1_score(test_row, test_output).score)

Run the experiment

Now you can run an experiment with this task, evaluator, and dataset.

experiment = arize_client.run_experiment(
    space_id=SPACE_ID,
    dataset_id=dataset_id,
    task=partial(summarize_article, prompt_template=BASE_TEMPLATE, model=MODEL),
    evaluators=EVALUATORS,
    experiment_name="initial-template",
)

Compare prompt variants

The point of holding the dataset and evaluators fixed is that any score change is attributable to the prompt. Run the same experiment again with a stricter instruction:

CONCISE_TEMPLATE = """
Summarize the article in two to four sentences. Be concise and include only the
most important information.

ARTICLE
=======
{article}

SUMMARY
=======
"""

arize_client.run_experiment(
    space_id=SPACE_ID,
    dataset_id=dataset_id,
    task=partial(summarize_article, prompt_template=CONCISE_TEMPLATE, model=MODEL),
    evaluators=EVALUATORS,
    experiment_name="concise-template",
)

Then try few-shot. The examples are drawn from the dataset’s train split, so they never overlap with the ten test rows you are scoring against.

EXAMPLE_TEMPLATE = """
ARTICLE
=======
{article}

SUMMARY
=======
{summary}
"""

train_df = (
    hf_ds["train"]
    .to_pandas()
    .sample(n=5, random_state=42)
    .rename(columns={"highlights": "summary"})
)
examples = "\n".join(
    EXAMPLE_TEMPLATE.format(article=row["article"], summary=row["summary"])
    for _, row in train_df.iterrows()
)

FEW_SHOT_TEMPLATE = (
    """
Summarize the article in two to four sentences. Be concise and include only the
most important information, as in the examples below.

EXAMPLES
========

"""
    + examples
    + """

Now summarize the following article.

ARTICLE
=======
{article}

SUMMARY
=======
"""
)

arize_client.run_experiment(
    space_id=SPACE_ID,
    dataset_id=dataset_id,
    task=partial(summarize_article, prompt_template=FEW_SHOT_TEMPLATE, model=MODEL),
    evaluators=EVALUATORS,
    experiment_name="few-shot-template",
)

Open the dataset’s Experiments tab to compare the three runs. Each experiment is a row and each evaluator its own score column, so you can read F1 against precision and recall together and see which prompt traded which for which.

Instrument

Observe

Evaluate

Improve

Advanced Workflows

Prompt Experimentation For Summarization Task

Before you start

Create a dataset

Define our experiment

Define our evaluators

Run the experiment

Compare prompt variants

​Before you start

​Create a dataset

​Define our experiment

​Define our evaluators

​Run the experiment

​Compare prompt variants

Before you start

Create a dataset

Define our experiment

Define our evaluators

Run the experiment

Compare prompt variants