Evaluations Quickstart

Offline evaluations are run as code and then sent back to Arize AX using client.spans.update_evaluations. This guide assumes you have traces in Arize AX and are looking to run an evaluation to measure your application performance. To add evaluations you can set up online evaluations as a task to run automatically, or you can follow the steps below to generate evaluations and log them to Arize AX:

Install the Arize SDK

Import your spans in code

Run a custom evaluator using Phoenix Evals

Log evaluations back to Arize AX

Install dependencies and setup keys

pip install 'arize>=8.0.0' arize-phoenix-evals openai pandas

Copy the ARIZE_API_KEY and SPACE_ID from your Space Settings page (shown below) and set them as environment variables alongside your OpenAI key.

export ARIZE_SPACE_ID="<your-space-id>"
export ARIZE_API_KEY="<your-api-key>"
export OPENAI_API_KEY="<your-openai-api-key>"

import os

SPACE_ID = os.environ["ARIZE_SPACE_ID"]
API_KEY = os.environ["ARIZE_API_KEY"]

Import your spans in code

Once you have traces in Arize AX, you can visit the LLM Tracing tab to see your traces and export them in code. By clicking the export button, you can get the boilerplate code to copy paste to your evaluator.

# import statements required for getting your spans
from datetime import datetime, timedelta, timezone
from arize import ArizeClient

start_time = datetime.now(timezone.utc) - timedelta(days=14)  # 14 days ago
end_time = datetime.now(timezone.utc)  # Today

# Exporting your spans into a dataframe
client = ArizeClient(api_key=API_KEY)
primary_df = client.spans.export_to_df(
    space_id=SPACE_ID,
    project_name="tracing-haiku-tutorial",  # change this to the name of your project
    start_time=start_time,
    end_time=end_time,
)

Run a custom evaluator using Phoenix Evals

Create a classifier for the LLM to judge the quality of your responses. You can utilize any of the Arize AX Evaluator Templates or you can create your own. Below is an example which judges the positivity or negativity of the LLM output. create_classifier relies on the judge’s tool-calling / structured-output support, so use a non-reasoning model such as GPT-4.1.

from phoenix.evals import LLM, create_classifier

judge = LLM(provider="openai", model="gpt-4.1")

MY_CUSTOM_TEMPLATE = """
    You are evaluating the positivity or negativity of the responses to questions.
    [BEGIN DATA]
    ************
    [Question]: {input}
    ************
    [Response]: {output}
    [END DATA]


    Please focus on the tone of the response.
    Your answer must be single word, either "positive" or "negative"
    """

tone_eval = create_classifier(
    name="tone_eval",
    prompt_template=MY_CUSTOM_TEMPLATE,
    llm=judge,
    choices={"positive": 1.0, "negative": 0.0},
    direction="maximize",
)

Notice the variables in brackets for and above. You will need to set those variables appropriately for the dataframe so you can run your custom template. We use OpenInference as a set of conventions (complementary to OpenTelemetry) to trace AI applications. This means depending on the provider you are using, the attributes of the trace will be different. You can use the code below to check which attributes are in the traces in your dataframe.

primary_df.columns

Use the code below to set the input and output variables needed for the prompt above.

primary_df["input"] = primary_df["attributes.input.value"]
primary_df["output"] = primary_df["attributes.output.value"]

Use the evaluate_dataframe function to run the evaluation using your classifier. You will be using the dataframe from the traces you generated above. It runs the judge calls concurrently and returns the dataframe with a tone_eval_score column holding the label, score, and explanation.

from phoenix.evals import evaluate_dataframe

results_df = evaluate_dataframe(dataframe=primary_df, evaluators=[tone_eval])

If you’d like more information, see our detailed guide on custom evaluators. You can also use our pre-tested evaluators for evaluating hallucination, toxicity, retrieval, etc.

Log evaluations back to Arize AX

Use the update_evaluations method on the Arize SDK client to attach the evaluations you’ve run to traces. It requires four columns, and the <eval_name> must be alphanumeric and cannot have hyphens or spaces.

eval.<eval_name>.label
eval.<eval_name>.score
eval.<eval_name>.explanation
context.span_id

to_annotation_dataframe flattens the nested tone_eval_score column into label, score, and explanation while preserving context.span_id (the join key export_to_df already provides). Rename those to the reserved eval.<eval_name>.* columns and upload:

from phoenix.evals.utils import to_annotation_dataframe

annotations = to_annotation_dataframe(dataframe=results_df)

evals_df = annotations.rename(
    columns={
        "label":       "eval.tone_eval.label",
        "score":       "eval.tone_eval.score",
        "explanation": "eval.tone_eval.explanation",
    }
)[[
    "context.span_id",
    "eval.tone_eval.label",
    "eval.tone_eval.score",
    "eval.tone_eval.explanation",
]]

# send the evals to Arize AX
client.spans.update_evaluations(
    space_id=SPACE_ID,
    project_name="tracing-haiku-tutorial",  # your project name
    dataframe=evals_df,
)

Instrument

Observe

Evaluate

Improve

Advanced Workflows

Install dependencies and setup keys

Import your spans in code

Run a custom evaluator using Phoenix Evals

Log evaluations back to Arize AX

​Install dependencies and setup keys

​Import your spans in code

​Run a custom evaluator using Phoenix Evals

​Log evaluations back to Arize AX

Install dependencies and setup keys

Import your spans in code

Run a custom evaluator using Phoenix Evals

Log evaluations back to Arize AX