MLflow

MLflow ships a GenAI evaluation API whose third-party scorers include the Phoenix evaluators (Hallucination, QA, Relevance, Toxicity, Summarization) from mlflow.genai.scorers.phoenix. Each scorer is a plain callable that returns an MLflow Feedback, so you can run it on Arize AX data without an MLflow tracking server. This guide shows both ways to wire MLflow scorers into Arize AX: Flow 1 grades existing Arize AX traces with the Hallucination scorer and writes the results back via client.spans.update_evaluations(...); Flow 2 uploads a small dataset, runs an Arize AX experiment with the same scorer as its evaluator, and surfaces the scores in Datasets+Experiments. Both flows share the same setup. Run the code blocks below in order inside a single Python session — each block builds on imports and variables from earlier ones.

Prerequisites

Python 3.11+
An ARIZE_SPACE_ID and ARIZE_API_KEY from your Arize AX space settings
An OPENAI_API_KEY from OpenAI Platform (used as both the model under trace and the MLflow scorer’s judge LLM)

Launch Arize AX

If you don’t already have an Arize AX account, sign up at arize.com and grab your ARIZE_SPACE_ID and ARIZE_API_KEY from Settings → Space Settings.

Install

# Pin arize-phoenix-evals<3: mlflow.genai.scorers.phoenix wraps the
# legacy Phoenix evaluator classes (HallucinationEvaluator, QAEvaluator,
# ...), which arize-phoenix-evals 3.0 removed. With 3.x installed every
# MLflow Phoenix scorer fails with "Unknown Phoenix metric".
pip install mlflow 'arize-phoenix-evals<3' 'arize>=8.0.0' openai \
  openinference-instrumentation-openai opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-grpc pandas

Configure credentials

export ARIZE_SPACE_ID="<your-space-id>"
export ARIZE_API_KEY="<your-api-key>"
export OPENAI_API_KEY="<your-openai-api-key>"

Define evaluators

The shared setup: an MLflow Hallucination scorer backed by GPT-4.1, the canonical 2-row hallucination dataset that both flows score, and an Arize SDK client. The scorer’s model argument is an MLflow model URI (<provider>:/<model>), so openai:/gpt-4.1 calls OpenAI with your OPENAI_API_KEY. The scorer runs entirely in-process — no MLflow tracking server or mlflow.start_run() is needed.

# combined.py
import os
import time
from datetime import datetime, timedelta, timezone

import pandas as pd
from arize import ArizeClient
from mlflow.genai.scorers.phoenix import Hallucination
from openai import OpenAI

SPACE_ID = os.environ["ARIZE_SPACE_ID"]
API_KEY = os.environ["ARIZE_API_KEY"]
TIMESTAMP = int(time.time())

# The Phoenix Hallucination evaluator, exposed through MLflow's scorer
# interface. It labels a response "factual" or "hallucinated" against the
# reference text passed in `expectations`.
hallucination = Hallucination(model="openai:/gpt-4.1")

# Canonical 2-row dataset — row 0 is factual (answer matches the reference),
# row 1 is hallucinated. Both flows grade these same rows.
ROWS = [
    {
        "input":     "What is the capital of France?",
        "output":    "Paris is the capital of France.",
        "reference": "Paris is the capital and most populous city of France.",
    },
    {
        "input":     "What is the capital of France?",
        "output":    "Berlin is the capital of France.",
        "reference": "Paris is the capital and most populous city of France.",
    },
]


def score_hallucination(question, answer, reference):
    """Run the MLflow scorer and map its Feedback to a 1.0/0.0 score."""
    feedback = hallucination(
        inputs=question,
        outputs=answer,
        expectations={"context": reference},
    )
    if feedback.error is not None:
        raise RuntimeError(f"MLflow scorer failed: {feedback.error}")
    is_factual = feedback.value == "factual"
    return (
        1.0 if is_factual else 0.0,
        "factual" if is_factual else "hallucinated",
        feedback.rationale or "no explanation",
    )


arize = ArizeClient(api_key=API_KEY)

Flow 1 — Evaluate existing traces

Source the spans

Instrument OpenAI with OpenInference, make two calls (each forced to echo a known answer so the trace contains predictable text), then pull the resulting spans back from Arize AX.

from openinference.instrumentation.openai import OpenAIInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

PROJECT_NAME = f"mlflow-tracing-example-{TIMESTAMP}"

resource = Resource.create(
    {
        "service.name":                PROJECT_NAME,
        "openinference.project.name":  PROJECT_NAME,
        "model_id":                    PROJECT_NAME,
    }
)
provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="https://otlp.arize.com:443",
            headers={
                "authorization":     API_KEY,
                "arize-space-id":    SPACE_ID,
                "arize-interface":   "python",
            },
        )
    )
)
trace.set_tracer_provider(provider)
OpenAIInstrumentor().instrument(tracer_provider=provider)

oai = OpenAI()
for row in ROWS:
    oai.chat.completions.create(
        model="gpt-6-luna",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a fact-recall assistant. The user states the "
                    "exact answer to use; reply with that verbatim."
                ),
            },
            {
                "role": "user",
                "content": (
                    f"Question: {row['input']}\n"
                    f"Answer (reply verbatim): {row['output']}"
                ),
            },
        ],
    )

provider.force_flush(timeout_millis=10_000)
print(f"Project: {PROJECT_NAME}")

# Spans take ~5–15s to be queryable after flush. Poll defensively: Arize's
# OTLP ingest and Flight export use different catalogs and the new project
# can briefly appear "unauthorized" to the export endpoint while still
# accepting span writes via OTLP, so swallow transient errors and retry.
start = datetime.now(timezone.utc) - timedelta(minutes=5)
end = datetime.now(timezone.utc) + timedelta(minutes=1)
spans_df = None
last_err: Exception | None = None
for _ in range(12):
    time.sleep(5)
    try:
        spans_df = arize.spans.export_to_df(
            space_id=SPACE_ID,
            project_name=PROJECT_NAME,
            start_time=start,
            end_time=end,
        )
    except Exception as e:
        last_err = e
        continue
    if spans_df is not None and len(spans_df) >= len(ROWS):
        break
else:
    raise RuntimeError(
        f"Spans never appeared after 60s (last error: {last_err})"
    )

spans_df = spans_df.sort_values("start_time").reset_index(drop=True)

Run the evaluators

Call the scorer once per span. It takes keyword arguments only: inputs for the question, outputs for the response, and expectations={"context": ...} for the reference text it grades against. The Feedback it returns carries the label in value and the judge’s reasoning in rationale.

scores = []
labels = []
explanations = []
for i, row in spans_df.iterrows():
    score, label, explanation = score_hallucination(
        ROWS[i]["input"],
        row["output"],
        ROWS[i]["reference"],
    )
    scores.append(score)
    labels.append(label)
    explanations.append(explanation)

Log evaluations to Arize AX

update_evaluations(...) requires a context.span_id column (which export_to_df already provides) plus the reserved eval.<name>.{score,label,explanation} columns. Each MLflow Feedback becomes one row in this DataFrame.

eval_df = pd.DataFrame(
    {
        "context.span_id":                 spans_df["context.span_id"],
        "eval.hallucination.score":        scores,
        "eval.hallucination.label":        labels,
        "eval.hallucination.explanation":  explanations,
    }
)
arize.spans.update_evaluations(
    space_id=SPACE_ID,
    project_name=PROJECT_NAME,
    dataframe=eval_df,
)

# Print the scores so they appear in stdout for verification.
flow1_display = pd.DataFrame(
    {
        "input":  [r["input"]  for r in ROWS],
        "output": [r["output"] for r in ROWS],
        "score":  scores,
    }
)
print("Flow 1 results:")
print(flow1_display.to_string())

Expected output

Flow 1 results:
                            input                            output  score
0  What is the capital of France?   Paris is the capital of France.    1.0
1  What is the capital of France?  Berlin is the capital of France.    0.0

Verify in Arize AX

Open the project named mlflow-tracing-example-<timestamp> (the value printed above) in your Arize AX space. Each ChatCompletion span now carries a hallucination annotation column showing the score, label, and explanation written by update_evaluations(...).

Flow 2 — Run an experiment

Create a dataset

The dataset is the same two rows. The space= / examples= kwarg names match the v8 SDK exactly (note: not space_id= and not dataframe=).

DATASET_NAME = f"mlflow-experiment-example-ds-{TIMESTAMP}"
dataset_df = pd.DataFrame(ROWS)
arize.datasets.create(
    name=DATASET_NAME,
    space=SPACE_ID,
    examples=dataset_df,
)
print(f"Dataset: {DATASET_NAME}")

Define the task

The task function receives the dataset row and returns whatever the experiment should grade. The parameter name must be one of input, output, metadata, or dataset_row — a single-arg task with an unrecognized name is bound to dataset_row by default. A real workflow would call an LLM here; this passthrough keeps the example deterministic.

def task(dataset_row):
    return dataset_row["output"]

Wrap the evaluators

The MLflow scorer is synchronous, so the experiment evaluator is a plain def. Return an EvaluationResult with score, label, and explanation populated — leaving any of those as None triggers unsupported cast from null to <type>: reserved column cannot be coerced to canonical type at upload time.

from arize.experiments.evaluators.types import EvaluationResult


def hallucination_eval(input, output, dataset_row) -> EvaluationResult:
    score, label, explanation = score_hallucination(
        dataset_row["input"],
        output if isinstance(output, str) else str(output),
        dataset_row["reference"],
    )
    return EvaluationResult(
        score=score,
        label=label,
        explanation=explanation,
    )

Run the experiment

EXPERIMENT_NAME = f"mlflow-experiment-example-{TIMESTAMP}"
experiment, runs_df = arize.experiments.run(
    space=SPACE_ID,
    name=EXPERIMENT_NAME,
    dataset=DATASET_NAME,
    task=task,
    evaluators={"hallucination": hallucination_eval},
)
print(f"Experiment: {EXPERIMENT_NAME}")
print("Flow 2 results:")
flow2_display = runs_df[
    ["output", "eval.hallucination.score", "eval.hallucination.label"]
].rename(
    columns={
        "eval.hallucination.score": "score",
        "eval.hallucination.label": "label",
    }
)
print(flow2_display.to_string())

Expected output

Flow 2 results:
                             output  score         label
0   Paris is the capital of France.    1.0       factual
1  Berlin is the capital of France.    0.0  hallucinated

Verify in Arize AX

Open the Datasets + Experiments tab in Arize AX. The dataset mlflow-experiment-example-ds-<timestamp> and the experiment mlflow-experiment-example-<timestamp> (names printed above) appear with one run per dataset row, each carrying the hallucination score and label columns.

Troubleshooting

Unknown Phoenix metric: 'Hallucination'. Could not find 'HallucinationEvaluator' in 'phoenix.evals'. You have arize-phoenix-evals 3.x installed. MLflow’s Phoenix scorers wrap the legacy evaluator classes that 3.0 removed. Reinstall with pip install 'arize-phoenix-evals<3'.
MLflow scorer failed: .... The scorer doesn’t raise on judge errors — it returns a Feedback whose error field is set, which score_hallucination turns into an exception. The message underneath is the judge call’s error; the most common cause is a missing or invalid OPENAI_API_KEY.
Scores look inverted. The underlying Phoenix evaluator scores hallucinated as 1 (it measures hallucination, not correctness) and puts that number in feedback.metadata["score"]. The page maps the label instead so that 1.0 means factual, matching the other evaluation integrations. Use the QA scorer if you’d rather grade correctness directly — it returns correct / incorrect.
column "eval.<name>.label": unsupported cast from null to string: reserved column cannot be coerced to canonical type. Your evaluator returned a bare number or string instead of a fully-populated EvaluationResult(score=..., label=..., explanation=...). Arize AX rejects null values in reserved eval columns — populate all three fields.
Spans never appear after 60s. Span flush + ingest typically takes 5–15s. If the loop times out, check that ARIZE_SPACE_ID + ARIZE_API_KEY are right and that you’re connecting to the correct region’s OTLP endpoint (otlp.arize.com for US, otlp.eu.arize.com for EU).
task failed for example id .... Your task function’s parameter name isn’t one of the recognized names (input, output, metadata, dataset_row). Rename it to dataset_row if you want the whole row, or pick the field you actually need.
Experiment runs duplicate or the dataset already exists. Both names embed TIMESTAMP = int(time.time()) so a single re-run produces unique names. If you re-execute the same combined.py quickly, regenerate TIMESTAMP first or call arize.experiments.delete(...) / arize.datasets.delete(...) on the prior run’s names.
Using a different judge model. Pass another MLflow model URI as model=, for example anthropic:/claude-sonnet-4-6 with ANTHROPIC_API_KEY set. MLflow calls providers it supports natively and falls back to LiteLLM for the rest, so install litellm if MLflow logs that it is falling back.

OpenTelemetry

LLM Providers

Agent Frameworks

Coding Agents

Platforms

Orchestration

Vector Databases

Evaluation

Prerequisites

Launch Arize AX

Install

Configure credentials

Define evaluators

Flow 1 — Evaluate existing traces

Source the spans

Run the evaluators

Log evaluations to Arize AX

Expected output

Verify in Arize AX

Flow 2 — Run an experiment

Create a dataset

Define the task

Wrap the evaluators

Run the experiment

Expected output

Verify in Arize AX

Troubleshooting

Resources

MLflow Phoenix scorers

MLflow on GitHub

Logging evaluations to Arize AX

Phoenix Evals with Arize AX

​Prerequisites

​Launch Arize AX

​Install

​Configure credentials

​Define evaluators

​Flow 1 — Evaluate existing traces

​Source the spans

​Run the evaluators

​Log evaluations to Arize AX

​Expected output

​Verify in Arize AX

​Flow 2 — Run an experiment

​Create a dataset

​Define the task

​Wrap the evaluators

​Run the experiment

​Expected output

​Verify in Arize AX

​Troubleshooting

​Resources

MLflow Phoenix scorers

MLflow on GitHub

Logging evaluations to Arize AX

Phoenix Evals with Arize AX

Prerequisites

Launch Arize AX

Install

Configure credentials

Define evaluators

Flow 1 — Evaluate existing traces

Source the spans

Run the evaluators

Log evaluations to Arize AX

Expected output

Verify in Arize AX

Flow 2 — Run an experiment

Create a dataset

Define the task

Wrap the evaluators

Run the experiment

Expected output

Verify in Arize AX

Troubleshooting

Resources