> ## Documentation Index > Fetch the complete documentation index at: /docs/llms.txt > Use this file to discover all available pages before exploring further. # MLflow > Use MLflow GenAI scorers backed by Phoenix evaluators to grade Arize AX traces and as evaluators in Arize AX experiments for LLM evaluation. [MLflow](https://mlflow.org/) ships a GenAI evaluation API whose third-party scorers include the [Phoenix evaluators](https://mlflow.org/docs/latest/genai/eval-monitor/scorers/third-party/phoenix.html) (`Hallucination`, `QA`, `Relevance`, `Toxicity`, `Summarization`) from `mlflow.genai.scorers.phoenix`. Each scorer is a plain callable that returns an MLflow `Feedback`, so you can run it on Arize AX data without an MLflow tracking server. This guide shows both ways to wire MLflow scorers into Arize AX: Flow 1 grades existing Arize AX traces with the `Hallucination` scorer and writes the results back via `client.spans.update_evaluations(...)`; Flow 2 uploads a small dataset, runs an Arize AX experiment with the same scorer as its evaluator, and surfaces the scores in Datasets+Experiments. Both flows share the same setup. Run the code blocks below in order inside a single Python session — each block builds on imports and variables from earlier ones. ## Prerequisites * Python 3.11+ * An `ARIZE_SPACE_ID` and `ARIZE_API_KEY` from your Arize AX space settings * An `OPENAI_API_KEY` from [OpenAI Platform](https://platform.openai.com/api-keys) (used as both the model under trace and the MLflow scorer's judge LLM) ## Launch Arize AX If you don't already have an Arize AX account, sign up at [arize.com](https://arize.com/) and grab your `ARIZE_SPACE_ID` and `ARIZE_API_KEY` from Settings → Space Settings. ## Install ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} # Pin arize-phoenix-evals<3: mlflow.genai.scorers.phoenix wraps the # legacy Phoenix evaluator classes (HallucinationEvaluator, QAEvaluator, # ...), which arize-phoenix-evals 3.0 removed. With 3.x installed every # MLflow Phoenix scorer fails with "Unknown Phoenix metric". pip install mlflow 'arize-phoenix-evals<3' 'arize>=8.0.0' openai \ openinference-instrumentation-openai opentelemetry-sdk \ opentelemetry-exporter-otlp-proto-grpc pandas ``` ## Configure credentials ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} export ARIZE_SPACE_ID="" export ARIZE_API_KEY="" export OPENAI_API_KEY="" ``` ## Define evaluators The shared setup: an MLflow `Hallucination` scorer backed by GPT-4.1, the canonical 2-row hallucination dataset that both flows score, and an Arize SDK client. The scorer's `model` argument is an MLflow model URI (`:/`), so `openai:/gpt-4.1` calls OpenAI with your `OPENAI_API_KEY`. The scorer runs entirely in-process — no MLflow tracking server or `mlflow.start_run()` is needed. ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} # combined.py import os import time from datetime import datetime, timedelta, timezone import pandas as pd from arize import ArizeClient from mlflow.genai.scorers.phoenix import Hallucination from openai import OpenAI SPACE_ID = os.environ["ARIZE_SPACE_ID"] API_KEY = os.environ["ARIZE_API_KEY"] TIMESTAMP = int(time.time()) # The Phoenix Hallucination evaluator, exposed through MLflow's scorer # interface. It labels a response "factual" or "hallucinated" against the # reference text passed in `expectations`. hallucination = Hallucination(model="openai:/gpt-4.1") # Canonical 2-row dataset — row 0 is factual (answer matches the reference), # row 1 is hallucinated. Both flows grade these same rows. ROWS = [ { "input": "What is the capital of France?", "output": "Paris is the capital of France.", "reference": "Paris is the capital and most populous city of France.", }, { "input": "What is the capital of France?", "output": "Berlin is the capital of France.", "reference": "Paris is the capital and most populous city of France.", }, ] def score_hallucination(question, answer, reference): """Run the MLflow scorer and map its Feedback to a 1.0/0.0 score.""" feedback = hallucination( inputs=question, outputs=answer, expectations={"context": reference}, ) if feedback.error is not None: raise RuntimeError(f"MLflow scorer failed: {feedback.error}") is_factual = feedback.value == "factual" return ( 1.0 if is_factual else 0.0, "factual" if is_factual else "hallucinated", feedback.rationale or "no explanation", ) arize = ArizeClient(api_key=API_KEY) ``` ## Flow 1 — Evaluate existing traces ### Source the spans Instrument OpenAI with OpenInference, make two calls (each forced to echo a known answer so the trace contains predictable text), then pull the resulting spans back from Arize AX. ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} from openinference.instrumentation.openai import OpenAIInstrumentor from opentelemetry import trace from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import ( OTLPSpanExporter, ) from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor PROJECT_NAME = f"mlflow-tracing-example-{TIMESTAMP}" resource = Resource.create( { "service.name": PROJECT_NAME, "openinference.project.name": PROJECT_NAME, "model_id": PROJECT_NAME, } ) provider = TracerProvider(resource=resource) provider.add_span_processor( BatchSpanProcessor( OTLPSpanExporter( endpoint="https://otlp.arize.com:443", headers={ "authorization": API_KEY, "arize-space-id": SPACE_ID, "arize-interface": "python", }, ) ) ) trace.set_tracer_provider(provider) OpenAIInstrumentor().instrument(tracer_provider=provider) oai = OpenAI() for row in ROWS: oai.chat.completions.create( model="gpt-6-luna", messages=[ { "role": "system", "content": ( "You are a fact-recall assistant. The user states the " "exact answer to use; reply with that verbatim." ), }, { "role": "user", "content": ( f"Question: {row['input']}\n" f"Answer (reply verbatim): {row['output']}" ), }, ], ) provider.force_flush(timeout_millis=10_000) print(f"Project: {PROJECT_NAME}") # Spans take ~5–15s to be queryable after flush. Poll defensively: Arize's # OTLP ingest and Flight export use different catalogs and the new project # can briefly appear "unauthorized" to the export endpoint while still # accepting span writes via OTLP, so swallow transient errors and retry. start = datetime.now(timezone.utc) - timedelta(minutes=5) end = datetime.now(timezone.utc) + timedelta(minutes=1) spans_df = None last_err: Exception | None = None for _ in range(12): time.sleep(5) try: spans_df = arize.spans.export_to_df( space_id=SPACE_ID, project_name=PROJECT_NAME, start_time=start, end_time=end, ) except Exception as e: last_err = e continue if spans_df is not None and len(spans_df) >= len(ROWS): break else: raise RuntimeError( f"Spans never appeared after 60s (last error: {last_err})" ) spans_df = spans_df.sort_values("start_time").reset_index(drop=True) ``` ### Run the evaluators Call the scorer once per span. It takes keyword arguments only: `inputs` for the question, `outputs` for the response, and `expectations={"context": ...}` for the reference text it grades against. The `Feedback` it returns carries the label in `value` and the judge's reasoning in `rationale`. ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} scores = [] labels = [] explanations = [] for i, row in spans_df.iterrows(): score, label, explanation = score_hallucination( ROWS[i]["input"], row["output"], ROWS[i]["reference"], ) scores.append(score) labels.append(label) explanations.append(explanation) ``` ### Log evaluations to Arize AX `update_evaluations(...)` requires a `context.span_id` column (which `export_to_df` already provides) plus the reserved `eval..{score,label,explanation}` columns. Each MLflow `Feedback` becomes one row in this DataFrame. ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} eval_df = pd.DataFrame( { "context.span_id": spans_df["context.span_id"], "eval.hallucination.score": scores, "eval.hallucination.label": labels, "eval.hallucination.explanation": explanations, } ) arize.spans.update_evaluations( space_id=SPACE_ID, project_name=PROJECT_NAME, dataframe=eval_df, ) # Print the scores so they appear in stdout for verification. flow1_display = pd.DataFrame( { "input": [r["input"] for r in ROWS], "output": [r["output"] for r in ROWS], "score": scores, } ) print("Flow 1 results:") print(flow1_display.to_string()) ``` ### Expected output ```text wrap theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} Flow 1 results: input output score 0 What is the capital of France? Paris is the capital of France. 1.0 1 What is the capital of France? Berlin is the capital of France. 0.0 ``` ### Verify in Arize AX Open the project named `mlflow-tracing-example-` (the value printed above) in your Arize AX space. Each `ChatCompletion` span now carries a `hallucination` annotation column showing the score, label, and explanation written by `update_evaluations(...)`. ## Flow 2 — Run an experiment ### Create a dataset The dataset is the same two rows. The `space=` / `examples=` kwarg names match the v8 SDK exactly (note: not `space_id=` and not `dataframe=`). ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} DATASET_NAME = f"mlflow-experiment-example-ds-{TIMESTAMP}" dataset_df = pd.DataFrame(ROWS) arize.datasets.create( name=DATASET_NAME, space=SPACE_ID, examples=dataset_df, ) print(f"Dataset: {DATASET_NAME}") ``` ### Define the task The task function receives the dataset row and returns whatever the experiment should grade. The parameter name **must** be one of `input`, `output`, `metadata`, or `dataset_row` — a single-arg task with an unrecognized name is bound to `dataset_row` by default. A real workflow would call an LLM here; this passthrough keeps the example deterministic. ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} def task(dataset_row): return dataset_row["output"] ``` ### Wrap the evaluators The MLflow scorer is synchronous, so the experiment evaluator is a plain `def`. Return an `EvaluationResult` with score, label, and explanation populated — leaving any of those as `None` triggers `unsupported cast from null to : reserved column cannot be coerced to canonical type` at upload time. ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} from arize.experiments.evaluators.types import EvaluationResult def hallucination_eval(input, output, dataset_row) -> EvaluationResult: score, label, explanation = score_hallucination( dataset_row["input"], output if isinstance(output, str) else str(output), dataset_row["reference"], ) return EvaluationResult( score=score, label=label, explanation=explanation, ) ``` ### Run the experiment ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} EXPERIMENT_NAME = f"mlflow-experiment-example-{TIMESTAMP}" experiment, runs_df = arize.experiments.run( space=SPACE_ID, name=EXPERIMENT_NAME, dataset=DATASET_NAME, task=task, evaluators={"hallucination": hallucination_eval}, ) print(f"Experiment: {EXPERIMENT_NAME}") print("Flow 2 results:") flow2_display = runs_df[ ["output", "eval.hallucination.score", "eval.hallucination.label"] ].rename( columns={ "eval.hallucination.score": "score", "eval.hallucination.label": "label", } ) print(flow2_display.to_string()) ``` ### Expected output ```text wrap theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} Flow 2 results: output score label 0 Paris is the capital of France. 1.0 factual 1 Berlin is the capital of France. 0.0 hallucinated ``` ### Verify in Arize AX Open the **Datasets + Experiments** tab in Arize AX. The dataset `mlflow-experiment-example-ds-` and the experiment `mlflow-experiment-example-` (names printed above) appear with one run per dataset row, each carrying the `hallucination` score and label columns. ## Troubleshooting * **`Unknown Phoenix metric: 'Hallucination'. Could not find 'HallucinationEvaluator' in 'phoenix.evals'`.** You have `arize-phoenix-evals` 3.x installed. MLflow's Phoenix scorers wrap the legacy evaluator classes that 3.0 removed. Reinstall with `pip install 'arize-phoenix-evals<3'`. * **`MLflow scorer failed: ...`.** The scorer doesn't raise on judge errors — it returns a `Feedback` whose `error` field is set, which `score_hallucination` turns into an exception. The message underneath is the judge call's error; the most common cause is a missing or invalid `OPENAI_API_KEY`. * **Scores look inverted.** The underlying Phoenix evaluator scores `hallucinated` as `1` (it measures hallucination, not correctness) and puts that number in `feedback.metadata["score"]`. The page maps the label instead so that `1.0` means factual, matching the other evaluation integrations. Use the `QA` scorer if you'd rather grade correctness directly — it returns `correct` / `incorrect`. * **`column "eval..label": unsupported cast from null to string: reserved column cannot be coerced to canonical type`.** Your evaluator returned a bare number or string instead of a fully-populated `EvaluationResult(score=..., label=..., explanation=...)`. Arize AX rejects null values in reserved eval columns — populate all three fields. * **Spans never appear after 60s.** Span flush + ingest typically takes 5–15s. If the loop times out, check that `ARIZE_SPACE_ID` + `ARIZE_API_KEY` are right and that you're connecting to the correct region's OTLP endpoint (`otlp.arize.com` for US, `otlp.eu.arize.com` for EU). * **`task failed for example id ...`.** Your task function's parameter name isn't one of the recognized names (`input`, `output`, `metadata`, `dataset_row`). Rename it to `dataset_row` if you want the whole row, or pick the field you actually need. * **Experiment runs duplicate or the dataset already exists.** Both names embed `TIMESTAMP = int(time.time())` so a single re-run produces unique names. If you re-execute the same `combined.py` quickly, regenerate `TIMESTAMP` first or call `arize.experiments.delete(...)` / `arize.datasets.delete(...)` on the prior run's names. * **Using a different judge model.** Pass another MLflow model URI as `model=`, for example `anthropic:/claude-sonnet-4-6` with `ANTHROPIC_API_KEY` set. MLflow calls providers it supports natively and falls back to LiteLLM for the rest, so install `litellm` if MLflow logs that it is falling back. ## Resources > **[MLflow Phoenix scorers](https://mlflow.org/docs/latest/genai/eval-monitor/scorers/third-party/phoenix.html)** > **[MLflow on GitHub](https://github.com/mlflow/mlflow)** > **[Logging evaluations to Arize AX](/docs/api-clients/python/version-8/client-resources/spans.md#update-evaluations)** > **[Phoenix Evals with Arize AX](/docs/ax/integrations/evaluation-integrations/phoenix-evals.md)**