Phoenix is an open-source observability tool designed for experimentation, evaluation, and troubleshooting of AI and LLM applications. It allows AI engineers and data scientists to quickly visualize their data, evaluate performance, track down issues, and export data to improve. Phoenix is built by Arize AI, the company behind the industry-leading AI observability platform, and a set of core contributors.

Phoenix works with OpenTelemetry and OpenInference instrumentation. See Integrations for details.

Features

Quickstarts

Running Phoenix for the first time? Select a quickstart below.

Next Steps

Try our Tutorials

Check out a comprehensive list of example notebooks for LLM Traces, Evals, RAG Analysis, and more.

Add Integrations

Add instrumentation for popular packages and libraries such as OpenAI, LangGraph, Vercel AI SDK and more.

Community

Join the Phoenix Slack community to ask questions, share findings, provide feedback, and connect with other developers.

Tracing can be augmented and customized by adding Metadata. Metadata includes your own custom attributes, user ids, session ids, prompt templates, and more.

• Learn how to add custom metadata and attributes to your traces

• Learn how to define custom prompt templates and variables in your tracing.

• Learn how to load a file of traces into Phoenix

• Learn how to export trace data from Phoenix

How to turn off tracing

Tracing can be paused temporarily or disabled permanently.

Pause tracing using context manager

If there is a section of your code for which tracing is not desired, e.g. the document chunking process, it can be put inside the suppress_tracing context manager as shown below.

from phoenix.trace import suppress_tracing

with suppress_tracing():
    # Code running inside this block doesn't generate traces.
    # For example, running LLM evals here won't generate additional traces.
    ...
# Tracing will resume outside the block.
...

Uninstrument the auto-instrumentors permanently

Calling .uninstrument() on the auto-instrumentors will remove tracing permanently. Below is the examples for LangChain, LlamaIndex and OpenAI, respectively.

LangChainInstrumentor().uninstrument()
LlamaIndexInstrumentor().uninstrument()
OpenAIInstrumentor().uninstrument()
# etc.

Prompt management allows you to create, store, and modify prompts for interacting with LLMs. By managing prompts systematically, you can improve reuse, consistency, and experiment with variations across different models and inputs.

Key benefits of prompt management include:

• Reusability: Store and load prompts across different use cases.

• Versioning: Track changes over time to ensure that the best performing version is deployed for use in your application.

• Collaboration: Share prompts with others to maintain consistency and facilitate iteration.

To learn how to get started with prompt management, see Create a prompt

Prompts in Phoenix can be created, iterated on, versioned, tagged, and used either via the UI or our Python/TS SDKs. The UI option also includes our Prompt Playground, which allows you to compare prompt variations side-by-side in the Phoenix UI.

Quickstarts

Sometimes while instrumenting your application, you may want to filter out or modify certain spans from being sent to Phoenix. For example, you may want to filter out spans that are that contain sensitive information or contain redundant information.

To do this, you can use a custom SpanProcessor and attach it to the OpenTelemetry TracerProvider.

Phoenix's tracing and span analysis capabilities are invaluable during the prototyping and debugging stages. By instrumenting application code with Phoenix, teams gain detailed insights into the execution flow, making it easier to identify and resolve issues. Developers can drill down into specific spans, analyze performance metrics, and access relevant logs and metadata to streamline debugging efforts.

This section contains details on Tracing features:

• Projects

• Annotations

• Sessions

Using Phoenix as a backend, Prompts can be managed and manipulated via code by using our Python or TypeScript SDKs.

With the Phoenix Client SDK you can:

• prompts dynamically

• templates by name, version, or tag

• templates with runtime variables and use them in your code. Native support for OpenAI, Anthropic, Gemini, Vercel AI SDK, and more. No propriatry client necessary.

• Support for and . Execute tools defined within the prompt. Phoenix prompts encompasses more than just the text and messages.

To learn more about managing Prompts in code, see

Testing in the Playground

Testing a prompt in the playground

The Playground is a fast and efficient way to refine prompt variations. You can load previous prompts and validate their performance by applying different variables.

Each single-run test in the Playground is recorded as a span in the Playground project, allowing you to revisit and analyze LLM invocations later. These spans can be added to datasets or reloaded for further testing.

Testing a prompt over a dataset

The ideal way to test a prompt is to construct a golden dataset where the dataset examples contains the variables to be applied to the prompt in the inputs and the outputs contains the ideal answer you want from the LLM. This way you can run a given prompt over N number of examples all at once and compare the synthesized answers against the golden answers.

Playground integrates with to help you iterate and incrementally improve your prompts. Experiment runs are automatically recorded and available for subsequent evaluation to help you understand how changes to your prompts, LLM model, or invocation parameters affect performance.

Testing prompt variations side-by-side

Prompt Playground supports side-by-side comparisons of multiple prompt variants. Click + Compare to add a new variant. Whether using Span Replay or testing prompts over a Dataset, the Playground processes inputs through each variant and displays the results for easy comparison.

Testing a prompt using code

Sometimes you may want to test a prompt and run evaluations on a given prompt. This can be particularly useful when custom manipulation is needed (e.x. you are trying to iterate on a system prompt on a variety of different chat messages). 🚧 This tutorial is coming soon

Sessions enable tracking and organizing related traces across multi-turn conversations with your AI application. When building conversational AI, maintaining context between interactions is critical - Sessions make this possible from an observability perspective.

With Sessions in Phoenix, you can:

• Track the entire history of a conversation in a single thread

• View conversations in a chatbot-like UI showing inputs and outputs of each turn

• Search through sessions to find specific interactions

• Track token usage and latency per conversation

This feature is particularly valuable for applications where context builds over time, like chatbots, virtual assistants, or any other multi-turn interaction. By tagging spans with a consistent session ID, you create a connected view that reveals how your application performs across an entire user journey.

Next Steps

• Check out how to

Projects provide organizational structure for your AI applications, allowing you to logically separate your observability data. This separation is essential for maintaining clarity and focus.

With Projects, you can:

• Segregate traces by environment (development, staging, production)

• Isolate different applications or use cases

• Track separate experiments without cross-contamination

• Maintain dedicated evaluation spaces for specific initiatives

• Create team-specific workspaces for collaborative analysis

Projects act as containers that keep related traces and conversations together while preventing them from interfering with unrelated work. This organization becomes increasingly valuable as you scale - allowing you to easily switch between contexts without losing your place or mixing data.

The Project structure also enables comparative analysis across different implementations, models, or time periods. You can run parallel versions of your application in separate projects, then analyze the differences to identify improvements or regressions.

Explore a Demo Project

Phoenix supports three main options to collect traces:

1. Use to mark functions and code blocks.

2. Use to capture all calls made to supported frameworks.

3. Use instrumentation. Supported in and

Quickstarts

Explore a Demo Trace

In order to improve your LLM application iteratively, it's vital to collect feedback, annotate data during human review, as well as to establish an evaluation pipeline so that you can monitor your application. In Phoenix we capture this type of feedback in the form of annotations.

Phoenix gives you the ability to annotate traces with feedback from the UI, your application, or wherever you would like to perform evaluation. Phoenix's annotation model is simple yet powerful - given an entity such as a span that is collected, you can assign a label and/or a score to that entity.

Next Steps

• Learn more about the concepts:

• Configure Annotation Configs to guide human annotations.

• How to run

• Learn how to log annotations via the client from your app or in a notebook

Prompt management allows you to create, store, and modify prompts for interacting with LLMs. By managing prompts systematically, you can improve reuse, consistency, and experiment with variations across different models and inputs.

Unlike traditional software, AI applications are non-deterministic and depend on natural language to provide context and guide model output. The pieces of natural language and associated model parameters embedded in your program are known as “prompts.”

Optimizing your prompts is typically the highest-leverage way to improve the behavior of your application, but “prompt engineering” comes with its own set of challenges. You want to be confident that changes to your prompts have the intended effect and don’t introduce regressions.

To get started, jump to .

Prompt Engineering Features

Phoenix offers a comprehensive suite of features to streamline your prompt engineering workflow.

• - Create, store, modify, and deploy prompts for interacting with LLMs

• - Play with prompts, models, invocation parameters and track your progress via tracing and experiments

• - Replay the invocation of an LLM. Whether it's an LLM step in an LLM workflow or a router query, you can step into the LLM invocation and see if any modifications to the invocation would have yielded a better outcome.

• - Phoenix offers client SDKs to keep your prompts in sync across different applications and environments.

Explore Demo Prompts

• Learn how to block PII from logging to Phoenix

• Learn how to selectively block or turn off tracing

• Learn how to send only certain spans to Phoenix

• Learn how to trace images

Span annotations can be an extremely valuable basis for improving your application. The Phoenix client provides useful ways to pull down spans and their associated annotations. This information can be used to:

• build new LLM judges

• form the basis for new datasets

• help identify ideas for improving your application

Pulling Spans

If you only want the spans that contain a specific annotation, you can pass in a query that filters on annotation names, scores, or labels.

The queries can also filter by annotation scores and labels.

This spans dataframe can be used to pull associated annotations.

Instead of an input dataframe, you can also pass in a list of ids:

The annotations and spans dataframes can be easily joined to produce a one-row-per-annotation dataframe that can be used to analyze the annotations!

Have you ever wanted to go back into a multi-step LLM chain and just replay one step to see if you could get a better outcome? Well you can with Phoenix's Span Replay. LLM spans that are stored within Phoenix can be loaded into the Prompt Playground and replayed. Replaying spans inside of Playground enables you to debug and improve the performance of your LLM systems by comparing LLM provider outputs, tweaking model parameters, changing prompt text, and more.

Chat completions generated inside of Playground are automatically instrumented, and the recorded spans are immediately available to be replayed inside of Playground.

The velocity of AI application development is bottlenecked by quality evaluations because AI engineers are often faced with hard tradeoffs: which prompt or LLM best balances performance, latency, and cost. High quality evaluations are critical as they can help developers answer these types of questions with greater confidence.

Datasets

Datasets are integral to evaluation. They are collections of examples that provide the inputs and, optionally, expected reference outputs for assessing your application. Datasets allow you to collect data from production, staging, evaluations, and even manually. The examples collected are used to run experiments and evaluations to track improvements to your prompt, LLM, or other parts of your LLM application.

Experiments

In AI development, it's hard to understand how a change will affect performance. This breaks the dev flow, making iteration more guesswork than engineering.

Experiments and evaluations solve this, helping distill the indeterminism of LLMs into tangible feedback that helps you ship more reliable product.

Specifically, good evals help you:

• Understand whether an update is an improvement or a regression

• Drill down into good / bad examples

• Compare specific examples vs. prior runs

• Avoid guesswork

Phoenix's Prompt Playground makes the process of iterating and testing prompts quick and easy. Phoenix's playground supports (OpenAI, Anthropic, Gemini, Azure) as well as custom model endpoints, making it the ideal prompt IDE for you to build experiment and evaluate prompts and models for your task.

• Speed: Rapidly test variations in the , model, invocation parameters, , and output format.

• Reproducibility: All runs of the playground are , unlocking annotations and evaluation.

• Datasets: Use as a fixture to run a prompt variant through its paces and to evaluate it systematically.

• Prompt Management: directly within the playground.

To learn more on how to use the playground, see

Moving your Phoenix deployment from development to production requires additional configuration for reliability, performance, and scalability. This page outlines the key steps and considerations to prepare your Phoenix instance for production workloads.

Configuration

Enable Batch Processing

Turn on the batch processor for spans, metrics, and logs. Batching improves data compression and reduces the number of outgoing connections required to transmit data efficiently. This is critical for stable ingestion at higher volumes.

The batch processor supports:

• Size-based batching (batch emits when a max number of items is reached)

• Time-based batching (batch emits after a configurable timeout)

Use gRPC Transport

Switch your exporters to use gRPC wherever possible to maximize payload compression and reduce network overhead in production environments.

Scaling Facilities

Plan for scaling resources to match your workload, including:

• Memory scaling for high-cardinality workloads or long retention windows.

• Disk scaling for log and trace ingestion, especially if retaining high volumes.

• Horizontal scaling if your deployment needs to handle increased concurrency.

Enable Database Backups

Ensure that automated backups are enabled for your Postgres instance. This protects your data and allows recovery in the event of failures or data corruption.

Resourcing Guidelines

Depending on your workload, you might need to provision your Phoenix instance with varying memory and data resources.

Memory Sizing

Memory requirements depend on several factors:

• Ingestion volume: Higher volumes of traces and logs increase memory needs for processing and indexing.

• Variety of labels and attributes: Workloads with many unique labels and attributes require additional memory for tracking and querying.

• Retention settings: Longer retention windows increase memory requirements for in-memory caching and indexing.

Monitor memory usage under expected production load and adjust resources to maintain your application performance.

Database Sizing

For production and scalable deployments, Phoenix supports PostgreSQL. The database size will depend on:

• Ingestion rate: Higher data ingestion will increase storage usage.

• Retention periods: Longer data retention requires additional storage capacity.

• Variety of labels and attributes: Workloads with many unique values consume more database space for indexing and storage.

Regularly monitor disk utilization to plan for scaling and ensure stable, reliable operation.

Backups

A solid backup plan protects your data and supports disaster recovery. Implement a Postgres backup strategy that considers:

• Backup frequency: How often backups occur.

• Backup methods: Such as point-in-time recovery (PITR) and full backups.

• Test restores: Regularly verify backups by restoring data.

Phoenix supports loading data that contains OpenInference traces. This allows you to load an existing dataframe of traces into your Phoenix instance.

Usually these will be traces you've previously saved using Save All Traces.

Connect to Phoenix

Before accessing px.Client(), be sure you've set the following environment variables:

import os

os.environ["PHOENIX_CLIENT_HEADERS"] = f"api_key=..."
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "https://app.phoenix.arize.com"

If you're self-hosting Phoenix, ignore the client headers and change the collector endpoint to your endpoint.

Importing Traces to an Existing Phoenix Instance

import phoenix as px

# Re-launch the app using trace data
px.launch_app(trace=px.TraceDataset(df))

# Load traces into an existing Phoenix instance
px.Client().log_traces(trace_dataset=px.TraceDataset(df))

# Load traces into an existing Phoenix instance from a local file
px.launch_app(trace=px.TraceDataset.load('f7733fda-6ad6-4427-a803-55ad2182b662', directory="/my_saved_traces/"))

Launching a new Phoenix Instance with Saved Traces

You can also launch a temporary version of Phoenix in your local notebook to quickly view the traces. But be warned, this Phoenix instance will only last as long as your notebook environment is runing

# Load traces from a dataframe
px.launch_app(trace=px.TraceDataset.load(my_traces))

# Load traces from a local file
px.launch_app(trace=px.TraceDataset.load('f7733fda-6ad6-4427-a803-55ad2182b662', directory="/my_saved_traces/"))

How to create datasets

Datasets are critical assets for building robust prompts, evals, fine-tuning, and much more. Phoenix allows you to build datasets manually, programmatically, or from files.

• Create datasets from CSV

• Create datasets from Pandas

• Create datasets from spans

• Create datasets using synthetic data

Exporting datasets

Export datasets for offline analysis, evals, and fine-tuning.

• Exporting to CSV - how to quickly download a dataset to use elsewhere

• Exporting to OpenAI Ft - want to fine tune an LLM for better accuracy and cost? Export llm examples for fine-tuning.

• Exporting to OpenAI Evals - have some good examples to use for benchmarking of llms using OpenAI evals? export to OpenAI evals format.

How to run experiments

• How to upload a Dataset

• How to run a custom task

• How to configure evaluators

• How to run the experiment

How to use evaluators

• LLM Evaluators

• Code Evaluators

• Custom Evaluators

Exporting to CSV

Want to just use the contents of your dataset in another context? Simply click on the export to CSV button on the dataset page and you are good to go!

Exporting for Fine-Tuning

Fine-tuning lets you get more out of the models available by providing:

• Higher quality results than prompting

• Ability to train on more examples than can fit in a prompt

• Token savings due to shorter prompts

• Lower latency requests

Fine-tuning improves on few-shot learning by training on many more examples than can fit in the prompt, letting you achieve better results on a wide number of tasks. Once a model has been fine-tuned, you won't need to provide as many examples in the prompt. This saves costs and enables lower-latency requests. Phoenix natively exports OpenAI Fine-Tuning JSONL as long as the dataset contains compatible inputs and outputs.

Exporting OpenAI Evals

Evals provide a framework for evaluating large language models (LLMs) or systems built using LLMs. OpenAI Evals offer an existing registry of evals to test different dimensions of OpenAI models and the ability to write your own custom evals for use cases you care about. You can also use your data to build private evals. Phoenix can natively export the OpenAI Evals format as JSONL so you can use it with OpenAI Evals. See https://github.com/openai/evals for details.

The Emotion Detection Eval Template is designed to classify emotions from audio files. This evaluation leverages predefined characteristics, such as tone, pitch, and intensity, to detect the most dominant emotion expressed in an audio input. This guide will walk you through how to use the template within the Phoenix framework to evaluate emotion classification models effectively.

Template Details

The following is the structure of the EMOTION_PROMPT_TEMPLATE:

You are an AI system designed to classify emotions in audio files.

### TASK:
Analyze the provided audio file and classify the primary emotion based on these characteristics:
- Tone: General tone of the speaker (e.g., cheerful, tense, calm).
- Pitch: Level and variability of the pitch (e.g., high, low, monotone).
- Pace: Speed of speech (e.g., fast, slow, steady).
- Volume: Loudness of the speech (e.g., loud, soft, moderate).
- Intensity: Emotional strength or expression (e.g., subdued, sharp, exaggerated).

The classified emotion must be one of the following:
['anger', 'happiness', 'excitement', 'sadness', 'neutral', 'frustration', 'fear', 'surprise', 'disgust', 'other']

IMPORTANT: Choose the most dominant emotion expressed in the audio. Neutral should only be used when no other emotion is clearly present; do your best to avoid this label.

************

Here is the audio to classify:

{audio}

RESPONSE FORMAT:

Provide a single word from the list above representing the detected emotion.

************

EXAMPLE RESPONSE: excitement

************

Analyze the audio and respond in this format.

Template Module

The prompt and evaluation logic are part of the phoenix.evals.default_audio_templates module and are defined as:

• EMOTION_AUDIO_RAILS: Output options for the evaluation template.

• EMOTION_PROMPT_TEMPLATE: Prompt used for evaluating audio emotions.

Use this prompt template to evaluate an agent's final response. This is an optional step, which you can use as a gate to retry a set of actions if the response or state of the world is insufficient for the given task.

Read more:

• How to evaluate the evaluators and build self-improving evals

• Prompt optimization

Prompt Template

This prompt template is heavily inspired by the paper: "Self Reflection in LLM Agents".

You are an expert in {topic}. I will give you a user query. Your task is to reflect on your provided solution and whether it has solved the problem.
First, explain whether you believe the solution is correct or incorrect.
Second, list the keywords that describe the type of your errors from most general to most specific.
Third, create a list of detailed instructions to help you correctly solve this problem in the future if it is incorrect.

Be concise in your response; however, capture all of the essential information.

Here is the data:
    [BEGIN DATA]
    ************
    [User Query]: {user_query}
    ************
    [Tools]: {tool_definitions}
    ************
    [State]: {current_state}
    ************
    [Provided Solution]: {solution}
    [END DATA]

The Phoenix app can be run in various environments such as Colab and SageMaker notebooks, as well as be served via the terminal or a docker container.

Phoenix Cloud

Phoenix Cloud provides free-to-use Phoenix instances that are preconfigured for you with 10GBs of storage space. Phoenix Cloud instances are a great starting point, however if you need more storage or more control over your instance, self-hosting options could be a better fit.

If you're using Phoenix Cloud, be sure to set the proper environment variables to connect to your instance:

import os

# Add Phoenix API Key for tracing
os.environ["PHOENIX_API_KEY"] = "ADD YOUR PHOENIX API KEY"
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "ADD YOUR PHOENIX HOSTNAME"

# If you created your Phoenix Cloud instance before June 24th, 2025,
# you also need to set the API key as a header
os.environ["PHOENIX_CLIENT_HEADERS"] = f"api_key={os.getenv('PHOENIX_API_KEY')}"

Container

See Self-Hosting.

Notebooks

To start phoenix in a notebook environment, run:

import phoenix as px

session = px.launch_app()

This will start a local Phoenix server. You can initialize the phoenix server with various kinds of data (traces, inferences).

Terminal

If you want to start a phoenix server to collect traces, you can also run phoenix directly from the command line:

phoenix serve

This will start the phoenix server on port 6006. If you are running your instrumented notebook or application on the same machine, traces should automatically be exported to http://127.0.0.1:6006 so no additional configuration is needed. However if the server is running remotely, you will have to modify the environment variable PHOENIX_COLLECTOR_ENDPOINT to point to that machine (e.g. http://<my-remote-machine>:<port>)

Phoenix traces AI applications, via OpenTelemetry and has first-class integrations with LlamaIndex, LangChain, OpenAI, and others.

LLM tracing records the paths taken by requests as they propagate through multiple steps or components of an LLM application. For example, when a user interacts with an LLM application, tracing can capture the sequence of operations, such as document retrieval, embedding generation, language model invocation, and response generation to provide a detailed timeline of the request's execution.

Tracing is a helpful tool for understanding how your LLM application works. Phoenix offers comprehensive tracing capabilities that are not tied to any specific LLM vendor or framework. Phoenix accepts traces over the OpenTelemetry protocol (OTLP) and supports first-class instrumentation for a variety of frameworks ( LlamaIndex, LangChain, DSPy), SDKs (OpenAI, Bedrock, Mistral, Vertex), and languages. (Python, JavaScript, etc.)

Using Phoenix's tracing capabilities can provide important insights into the inner workings of your LLM application. By analyzing the collected trace data, you can identify and address various performance and operational issues and improve the overall reliability and efficiency of your system.

• Application Latency: Identify and address slow invocations of LLMs, Retrievers, and other components within your application, enabling you to optimize performance and responsiveness.

• Token Usage: Gain a detailed breakdown of token usage for your LLM calls, allowing you to identify and optimize the most expensive LLM invocations.

• Runtime Exceptions: Capture and inspect critical runtime exceptions, such as rate-limiting events, that can help you proactively address and mitigate potential issues.

• Retrieved Documents: Inspect the documents retrieved during a Retriever call, including the score and order in which they were returned to provide insight into the retrieval process.

• Embeddings: Examine the embedding text used for retrieval and the underlying embedding model to allow you to validate and refine your embedding strategies.

• LLM Parameters: Inspect the parameters used when calling an LLM, such as temperature and system prompts, to ensure optimal configuration and debugging.

• Prompt Templates: Understand the prompt templates used during the prompting step and the variables that were applied, allowing you to fine-tune and improve your prompting strategies.

• Tool Descriptions: View the descriptions and function signatures of the tools your LLM has been given access to in order to better understand and control your LLM’s capabilities.

• LLM Function Calls: For LLMs with function call capabilities (e.g., OpenAI), you can inspect the function selection and function messages in the input to the LLM, further improving your ability to debug and optimize your application.

By using tracing in Phoenix, you can gain increased visibility into your LLM application, empowering you to identify and address performance bottlenecks, optimize resource utilization, and ensure the overall reliability and effectiveness of your system.

Next steps

• To get started, check out the Quickstart guide.

• Read more about and .

• Check out the How-To Guides for specific tutorials.

Instrumenting prompt templates and variables allows you to track and visualize prompt changes. These can also be combined with Experiments to measure the performance changes driven by each of your prompts.

@using_prompt_template(
    template=prompt_template,
    variables=prompt_template_variables,
    version="v1.0",
)
def call_fn(*args, **kwargs):
    # Calls within this function will generate spans with the attributes:
    # "llm.prompt_template.template" = "Please describe the weather forecast for {city} on {date}"
    # "llm.prompt_template.version" = "v1.0"
    # "llm.prompt_template.variables" = "{\"city\": \"Johannesburg\", \"date\": \"July 11\"}" # JSON serialized
    ...

import { context } from "@opentelemetry/api"
import { setPromptTemplate } from "@openinference-core"

context.with(
  setPromptTemplate(
    context.active(),
    { 
      template: "hello {{name}}",
      variables: { name: "world" },
      version: "v1.0"
    }
  ),
  () => {
      // Calls within this block will generate spans with the attributes:
      // "llm.prompt_template.template" = "hello {{name}}"
      // "llm.prompt_template.version" = "v1.0"
      // "llm.prompt_template.variables" = '{ "name": "world" }'
  }
)

Annotating traces is a crucial aspect of evaluating and improving your LLM-based applications. By systematically recording qualitative or quantitative feedback on specific interactions or entire conversation flows, you can:

1. Track performance over time

2. Identify areas for improvement

3. Compare different model versions or prompts

4. Gather data for fine-tuning or retraining

5. Provide stakeholders with concrete metrics on system effectiveness

Phoenix allows you to annotate traces through the Client, the REST API, or the UI.

Guides

• To learn how to configure annotations and to annotate through the UI, see Annotating in the UI

• To learn how to add human labels to your traces, either manually or programmatically, see Annotating via the Client

• To learn how to evaluate traces captured in Phoenix, see Running Evals on Traces

• To learn how to upload your own evaluation labels into Phoenix, see Log Evaluation Results

For more background on the concept of annotations, see Annotations

Overview

Phoenix supports three main options to collect traces:

1. Use Phoenix's decorators to mark functions and code blocks.

2. Use automatic instrumentation to capture all calls made to supported frameworks.

3. Use base OpenTelemetry instrumentation. Supported in Python and TS / JS, among many other languages.

This example uses options 1 and 2.

Launch Phoenix

Connect to Phoenix

To collect traces from your application, you must configure an OpenTelemetry TracerProvider to send traces to Phoenix.

pip install arize-phoenix-otel

from phoenix.otel import register

# configure the Phoenix tracer
tracer_provider = register(
  project_name="my-llm-app", # Default is 'default'
  auto_instrument=True, # See 'Trace all calls made to a library' below
)
tracer = tracer_provider.get_tracer(__name__)

Trace your own functions

Functions can be traced using decorators:

@tracer.chain
def my_func(input: str) -> str:
    return "output"

Input and output attributes are set automatically based on my_func's parameters and return.

Trace all calls made to a library

Phoenix can also capture all calls made to supported libraries automatically. Just install the respective OpenInference library:

pip install openinference-instrumentation-openai

# Add OpenAI API Key
import os
import openai

os.environ["OPENAI_API_KEY"] = "ADD YOUR OPENAI API KEY"

client = openai.OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Write a haiku."}],
)
print(response.choices[0].message.content)

View your Traces in Phoenix

You should now see traces in Phoenix!

Next Steps

• Explore tracing integrations

• Customize tracing

• View use cases to see end-to-end examples

Getting Started

• Configure AI Providers - how to configure API keys for OpenAI, Anthropic, Gemini, and more.

Prompt Management

Organize and manage prompts with Phoenix to streamline your development workflow

• Create a prompt - how to create, update, and track prompt changes

• Test a prompt - how to test changes to a prompt in the playground and in the notebook

• Tag a prompt - how to mark certain prompt versions as ready for production

• Using a prompt - how to integrate prompts into your code and experiments

Playground

Iterate on prompts and models in the prompt playground

• Using the Playground - how to setup the playground and how to test prompt changes via datasets and experiments.

The standard for evaluating text is human labeling. However, high-quality LLM outputs are becoming cheaper and faster to produce, and human evaluation cannot scale. In this context, evaluating the performance of LLM applications is best tackled by using a LLM. The Phoenix LLM Evals library is designed for simple, fast, and accurate LLM-based evaluations.

Phoenix Evals come with:

• Pre-built evals - Phoenix provides pre-tested eval templates for common tasks such as RAG and function calling. Learn more about pretested templates here. Each eval is pre-tested on a variety of eval models. Find the most up-to-date template on GitHub.

• Run evals on your own data - Phoenix Evals takes a dataframe as its primary input and output, making it easy to run evaluations on your own data - whether that's logs, traces, or datasets downloaded for benchmarking.

• Speed - Phoenix evals are designed for maximum speed and throughput. Evals run in batches and typically run 10x faster than calling the APIs directly.

• Built-in Explanations - All Phoenix evaluations include an explanation flag that requires eval models to explain their judgment rationale. This boosts performance and helps you understand and improve your eval.

• Eval Models - Phoenix lets you configure which foundation model you'd like to use as a judge. This includes OpenAI, Anthropic, Gemini, and much more. See Eval Models

Configuring Annotations

To annotate data in the UI, you first will want to setup a rubric for how to annotate. Navigate to Settings and create annotation configs (e.g. a rubric) for your data. You can create various different types of annotations: Categorical, Continuous, and Freeform.

Adding Annotations

Once you have annotations configured, you can associate annotations to the data that you have traced. Click on the Annotate button and fill out the form to rate different steps in your AI application. You can also take notes as you go by either clicking on the explain link or by adding your notes to the bottom messages UI. You can always come back and edit / and delete your annotations. Annotations can be deleted from the table view under the Annotations tab.

Viewing Annotations

As annotations come in from various sources (annotators, evals), the entire list of annotations can be found under the Annotations tab. Here you can see the author, the annotator kind (e.g. was the annotation performed by a human, llm, or code), and so on. This can be particularly useful if you want to see if different annotators disagree.

Exporting Traces with specific Annotation Values

Once you have collected feedback in the form of annotations, you can filter your traces by the annotation values to narrow down to interesting samples (e.x. llm spans that are incorrect). Once filtered down to a sample of spans, you can export your selection to a dataset, which in turn can be used for things like experimentation, fine-tuning, or building a human-aligned eval.

You can use cron to run evals client-side as your traces and spans are generated, augmenting your dataset with evaluations in an online manner. View the example in Github.

This example:

• Continuously queries a LangChain application to send new traces and spans to your Phoenix session

• Queries new spans once per minute and runs evals, including:

  • Hallucination

  • Q&A Correctness

  • Relevance

• Logs evaluations back to Phoenix so they appear in the UI

The evaluation script is run as a cron job, enabling you to adjust the frequency of the evaluation job:

* * * * * /path/to/python /path/to/run_evals.py

The above script can be run periodically to augment Evals in Phoenix.

When your agents take multiple steps to get to an answer or resolution, it's important to evaluate the pathway it took to get there. You want most of your runs to be consistent and not take unnecessarily frivolous or wrong actions.

One way of doing this is to calculate convergence:

1. Run your agent on a set of similar queries

2. Record the number of steps taken for each

3. Calculate the convergence score: avg(minimum steps taken / steps taken for this run)

This will give a convergence score of 0-1, with 1 being a perfect score.

# Assume you have an output which has a list of messages, which is the path taken
all_outputs = [
]

optimal_path_length = min(all_outputs, key = lambda output: len(output))
ratios_sum = 0

for output in all_outputs:
    run_length = len(output)
    ratio = optimal_path_length / run_length
    ratios_sum += ratio

# Calculate the average ratio
if len(all_outputs) > 0:
    convergence = ratios_sum / len(all_outputs)
else:
    convergence = 0

print(f"The optimal path length is {optimal_path_length}")
print(f"The convergence is {convergence}")

Log to a specific project

Phoenix uses projects to group traces. If left unspecified, all traces are sent to a default project.

Projects work by setting something called the Resource attributes (as seen in the OTEL example above). The phoenix server uses the project name attribute to group traces into the appropriate project.

Switching projects in a notebook

Typically you want traces for an LLM app to all be grouped in one project. However, while working with Phoenix inside a notebook, we provide a utility to temporarily associate spans with different projects. You can use this to trace things like evaluations.

Phoenix supports displaying images that are included in LLM traces.

To view images in Phoenix

3. Include either a base64 UTF-8 encoded image or an image url in the call made to your LLM

Example

You should see your image appear in Phoenix:

• (llm_classify)

• (llm_generate)

Run evaluations via a job to visualize in the UI as traces stream in.

Evaluate traces captured in Phoenix and export results to the Phoenix UI.

Evaluate tasks with multiple inputs/outputs (ex: text, audio, image) using versatile evaluation tasks.

This template evaluates a plan generated by an agent. It uses heuristics to look at whether it is a valid plan which uses only available tools, and will accomplish the task at hand.

Prompt Template

Setup

To first get started, you will first . In the playground view, create a valid prompt for the LLM and click Run on the top right (or the mod + enter)

If successful you should see the LLM output stream out in the Output section of the UI.

Prompt Editor

The prompt editor (typically on the left side of the screen) is where you define the . You select the template language (mustache or f-string) on the toolbar. Whenever you type a variable placeholder in the prompt (say {{question}} for mustache), the variable to fill will show up in the inputs section. Input variables must either be filled in by hand or can be filled in via a dataset (where each row has key / value pairs for the input).

Model Configuration

Every prompt instance can be configured to use a specific LLM and set of invocation parameters. Click on the model configuration button at the top of the prompt editor and configure your LLM of choice. Click on the "save as default" option to make your configuration sticky across playground sessions.

Comparing Prompts

The Prompt Playground offers the capability to compare multiple prompt variants directly within the playground. Simply click the + Compare button at the top of the first prompt to create duplicate instances. Each prompt variant manages its own independent template, model, and parameters. This allows you to quickly compare prompts (labeled A, B, C, and D in the UI) and run experiments to determine which prompt and model configuration is optimal for the given task.

Using Datasets with Prompts

Phoenix lets you run a prompt (or multiple prompts) on a dataset. Simply containing the input variables you want to use in your prompt template. When you click Run, Phoenix will apply each configured prompt to every example in the dataset, invoking the LLM for all possible prompt-example combinations. The result of your playground runs will be tracked as an experiment under the loaded dataset (see )

Playground Traces

All invocations of an LLM via the playground is recorded for analysis, annotations, evaluations, and dataset curation.

If you simply run an LLM in the playground using the free form inputs (e.g. not using a dataset), Your spans will be recorded in a project aptly titled "playground".

If however you run a prompt over dataset examples, the outputs and spans from your playground runs will be captured as an experiment. Each experiment will be named according to the prompt you ran the experiment over.

Many LLM applications use a technique called Retrieval Augmented Generation. These applications retrieve data from their knowledge base to help the LLM accomplish tasks with the appropriate context.

However, these retrieval systems can still hallucinate or provide answers that are not relevant to the user's input query. We can evaluate retrieval systems by checking for:

1. Are there certain types of questions the chatbot gets wrong more often?

2. Are the documents that the system retrieves irrelevant? Do we have the right documents to answer the question?

3. Does the response match the provided documents?

Phoenix supports retrievals troubleshooting and evaluation on both traces and inferences, but inferences are currently required to visualize your retrievals using a UMAP. See below on the differences.

Check out our to get started. Look at our to better understand how to troubleshoot and evaluate different kinds of retrieval systems. For a high level overview on evaluation, check out our .

Teams that are using conversation bots and assistants desire to know whether a user interacting with the bot is frustrated. The user frustration evaluation can be used on a single back and forth or an entire span to detect whether a user has become frustrated by the conversation.

User Frustration Eval Template

The following is an example of code snippet showing how to use the eval above template:

SQL Generation is a common approach to using an LLM. In many cases the goal is to take a human description of the query and generate matching SQL to the human description.

Example of a Question: How many artists have names longer than 10 characters?

Example Query Generated:

SELECT COUNT(ArtistId) \nFROM artists \nWHERE LENGTH(Name) > 10

The goal of the SQL generation Evaluation is to determine if the SQL generated is correct based on the question asked.

SQL Eval Template

Running an SQL Generation Eval

How to export your data for labeling, evaluation, or fine-tuning

phoenix.otel is a lightweight wrapper around OpenTelemetry primitives with Phoenix-aware defaults.

pip install arize-phoenix-otel

These defaults are aware of environment variables you may have set to configure Phoenix:

• PHOENIX_COLLECTOR_ENDPOINT

• PHOENIX_PROJECT_NAME

• PHOENIX_CLIENT_HEADERS

• PHOENIX_API_KEY

• PHOENIX_GRPC_PORT

Quickstart: phoenix.otel.register

The phoenix.otel module provides a high-level register function to configure OpenTelemetry tracing by setting a global TracerProvider. The register function can also configure headers and whether or not to process spans one by one or by batch.

from phoenix.otel import register
tracer_provider = register(
    project_name="default", # sets a project name for spans
    batch=True, # uses a batch span processor
    auto_instrument=True, # uses all installed OpenInference instrumentors
)

Phoenix Authentication

If the PHOENIX_API_KEY environment variable is set, register will automatically add an authorization header to each span payload.

Configuring the collector endpoint

There are two ways to configure the collector endpoint:

• Using environment variables

• Using the endpoint keyword argument

Using environment variables

If you're setting the PHOENIX_COLLECTOR_ENDPOINT environment variable, register will automatically try to send spans to your Phoenix server using gRPC.

# export PHOENIX_COLLECTOR_ENDPOINT=https://your-phoenix.com:6006

from phoenix.otel import register

# sends traces to https://your-phoenix.com:4317
tracer_provider = register()

# export PHOENIX_COLLECTOR_ENDPOINT=https://your-phoenix.com:6006

from phoenix.otel import register

# sends traces to https://your-phoenix.com/v1/traces
tracer_provider = register(
    protocol="http/protobuf",
)

Specifying the endpoint directly

When passing in the endpoint argument, you must specify the fully qualified endpoint. If the PHOENIX_GRPC_PORT environment variable is set, it will override the default gRPC port.

from phoenix.otel import register
tracer_provider = register(endpoint="http://localhost:6006/v1/traces")

from phoenix.otel import register
tracer_provider = register(endpoint="http://localhost:4317")

from phoenix.otel import register
tracer_provider = register(
    endpoint="http://localhost:9999",
    protocol="grpc", # use "http/protobuf" for http transport
)

Additional configuration

register can be configured with different keyword arguments:

• project_name: The Phoenix project name

  • or use PHOENIX_PROJECT_NAME env. var

• headers: Headers to send along with each span payload

  • or use PHOENIX_CLIENT_HEADERS env. var

• batch: Whether or not to process spans in batch

from phoenix.otel import register
tracer_provider = register(
    project_name="otel-test",
    headers={"Authorization": "Bearer TOKEN"},
    batch=True,
)

Instrumentation

Once you've connected your application to your Phoenix instance using phoenix.otel.register, you need to instrument your application. You have a few options to do this:

1. Using OpenInference auto-instrumentors. If you've used the auto_instrument flag above, then any instrumentor packages in your environment will be called automatically. For a full list of OpenInference packages, see https://arize.com/docs/phoenix/integrations

2. Using Phoenix Decorators.

3. Using Base OTEL.

When working with spans that are automatically instrumented via OpenInference in your LLM applications, you often need to capture span contexts to apply feedback or annotations. The capture_span_context context manager provides a convenient way to capture all OpenInference spans within its scope, making it easier to apply feedback to specific spans in downstream operations.

The capture_span_context context manager allows you to:

• Capture all spans created within a specific code block

• Retrieve span contexts for later use in feedback systems

• Maintain a clean separation between span creation and annotation logic

• Apply feedback to spans without needing to track span IDs manually

Usage

You can use the captured span contexts to implement custom feedback logic. The captured span contexts integrate seamlessly with Phoenix's annotation system:

from openinference.instrumentation import capture_span_context
from opentelemetry.trace.span import format_span_id
from phoenix.client import Client

client = Client()

def process_llm_request_with_feedback(prompt: str):
    with capture_span_context() as capture:
        # Make LLM call (auto-instrumented)
        response = llm.invoke("Generate a summary")
        # Get user feedback (simulated)
        user_feedback = get_user_feedback(response)
        
        # Method 1: Get span ID using get_last_span_id (most recent span)
        last_span_id = capture.get_last_span_id()
        # Apply feedback to the most recent span
        if last_span_id:
            client.annotations.add_span_annotation(
                annotation_name="user_feedback",
                annotator_kind="HUMAN",
                span_id=last_span_id,
                label=user_feedback.label,
                score=user_feedback.score,
                explanation=user_feedback.explanation
            )
        
        # Method 2: Get all captured span contexts and iterate
        span_contexts = capture.get_span_contexts()
        # Apply feedback to all captured spans
        for span_context in span_contexts:
            # Convert span context to span ID for annotation
            span_id = format_span_id(span_context.span_id)
            
            # Add annotation to Phoenix
            client.annotations.add_span_annotation(
                annotation_name="user_feedback_all",
                annotator_kind="HUMAN",
                span_id=span_id,
                label=user_feedback.label,
                score=user_feedback.score,
                explanation=user_feedback.explanation
            )

Working with Multiple Span Types

You can filter spans based on their attributes:

with capture_span_context() as capture:
    # Make LLM call (auto-instrumented)
    response = llm.invoke("Generate a summary")
    
    span_contexts = capture.get_span_contexts()
    
    # Filter for specific span types
    llm_spans = [
        ctx for ctx in span_contexts 
        if hasattr(ctx, 'attributes')
    ]
    
    # Apply different feedback logic to different span types
    for span_context in llm_spans:
        apply_llm_feedback(span_context)

Resources

• OpenInference

In some situations, you may need to modify the observability level of your tracing. For instance, you may want to keep sensitive information from being logged for security reasons, or you may want to limit the size of the base64 encoded images logged to reduced payload size.

The OpenInference Specification defines a set of environment variables you can configure to suit your observability needs. In addition, the OpenInference auto-instrumentors accept a trace config which allows you to set these value in code without having to set environment variables, if that's what you prefer

The possible settings are:

To set up this configuration you can either:

• Set environment variables as specified above

• Define the configuration in code as shown below

• Do nothing and fall back to the default values

• Use a combination of the three, the order of precedence is:

  • Values set in the TraceConfig in code

  • Environment variables

  • default values

Below is an example of how to set these values in code using our OpenAI Python and JavaScript instrumentors, however, the config is respected by all of our auto-instrumentors.

from openinference.instrumentation import TraceConfig
config = TraceConfig(        
    hide_inputs=...,
    hide_outputs=...,
    hide_input_messages=...,
    hide_output_messages=...,
    hide_input_images=...,
    hide_input_text=...,
    hide_output_text=...,
    hide_embedding_vectors=...,
    hide_llm_invocation_parameters=...,
    hide_llm_prompts=...,
    base64_image_max_length=...,
)

from openinference.instrumentation.openai import OpenAIInstrumentor
OpenAIInstrumentor().instrument(
    tracer_provider=tracer_provider,
    config=config,
)

/**
 * Everything left out of here will fallback to
 * environment variables then defaults
 */
const traceConfig = { hideInputs: true } 

const instrumentation = new OpenAIInstrumentation({ traceConfig })

Phoenix is a comprehensive platform designed to enable observability across every layer of an LLM-based system, empowering teams to build, optimize, and maintain high-quality applications and agents efficiently.

🛠️ Develop

During the development phase, Phoenix offers essential tools for debugging, experimentation, evaluation, prompt tracking, and search and retrieval.

🧪 Testing/Staging

In the testing and staging environment, Phoenix supports comprehensive evaluation, benchmarking, and data curation. Traces, experimentation, prompt tracking, and embedding visualizer remain important in the testing and staging phase, helping teams identify and resolve issues before deployment.

🚀 Production

In production, Phoenix works hand-in-hand with Arize, which focuses on the production side of the LLM lifecycle. The integration ensures a smooth transition from development to production, with consistent tooling and metrics across both platforms.

Phoenix natively integrates with OpenAI, Azure OpenAI, Anthropic, and Google AI Studio (gemini) to make it easy to test changes to your prompts. In addition to the above, since many AI providers (deepseek, ollama) can be used directly with the OpenAI client, you can talk to any OpenAI compatible LLM provider.

Credentials

To securely provide your API keys, you have two options. One is to store them in your browser in local storage. Alternatively, you can set them as environment variables on the server side. If both are set at the same time, the credential set in the browser will take precedence.

Option 1: Store API Keys in the Browser

API keys can be entered in the playground application via the API Keys dropdown menu. This option stores API keys in the browser. Simply navigate to to settings and set your API keys.

Option 2: Set Environment Variables on Server Side

Available on self-hosted Phoenix

If the following variables are set in the server environment, they'll be used at API invocation time.

Using OpenAI Compatible LLMs

Option 1: Configure the base URL in the prompt playground

Since you can configure the base URL for the OpenAI client, you can use the prompt playground with a variety of OpenAI Client compatible LLMs such as Ollama, DeepSeek, and more.\

OpenAI Client compatible providers Include

Option 2: Server side configuration of the OpenAI base URL

Optionally, the server can be configured with the OPENAI_BASE_URL environment variable to change target any OpenAI compatible REST API.

Installation

Start out by installing the Phoenix library:

Connect to Phoenix

You'll need to specify your Phoenix endpoint before you can interact with the Client. The easiest way to do this is through an environment variable.

Creating a Prompt

Now you can create a prompt. In this example, you'll create a summarization Prompt.

Prompts in Phoenix have names, as well as multiple versions. When you create your prompt, you'll define its name. Then, each time you update your prompt, that will create a new version of the prompt under the same name.

Your prompt will now appear in your Phoenix dashboard:

Retrieving a Prompt

You can retrieve a prompt by name, tag, or version:

Using a Prompt

To use a prompt, call the prompt.format()function. Any {{ variables }} in the prompt can be set by passing in a dictionary of values.

Updating a Prompt

To update a prompt with a new version, simply call the create function using the existing prompt name:

The new version will appear in your Phoenix dashboard:

Congratulations! You can now create, update, access and use prompts using the Phoenix SDK!

Next Steps

From here, check out:

• How to use your prompts in

• Prompt iteration

An evaluation must have a name (e.g. "Q&A Correctness") and its DataFrame must contain identifiers for the subject of evaluation, e.g. a span or a document (more on that below), and values under either the score, label, or explanation columns. See for more information.

Connect to Phoenix

Before accessing px.Client(), be sure you've set the following environment variables:

Span Evaluations

A dataframe of span evaluations would look similar like the table below. It must contain span_id as an index or as a column. Once ingested, Phoenix uses the span_id to associate the evaluation with its target span.

The evaluations dataframe can be sent to Phoenix as follows. Note that the name of the evaluation must be supplied through the eval_name= parameter. In this case we name it "Q&A Correctness".

Document Evaluations

A dataframe of document evaluations would look something like the table below. It must contain span_id and document_position as either indices or columns. document_position is the document's (zero-based) index in the span's list of retrieved documents. Once ingested, Phoenix uses the span_id and document_position to associate the evaluation with its target span and document.

The evaluations dataframe can be sent to Phoenix as follows. Note that the name of the evaluation must be supplied through the eval_name= parameter. In this case we name it "Relevance".

Logging Multiple Evaluation DataFrames

Multiple sets of Evaluations can be logged by the same px.Client().log_evaluations() function call.

Specifying A Project for the Evaluations

By default the client will push traces to the project specified in the PHOENIX_PROJECT_NAME environment variable or to the default project. If you want to specify the destination project explicitly, you can pass the project name as a parameter.

This guide will walk you through the process of evaluating traces captured in Phoenix, and exporting the results to the Phoenix UI.

This process is similar to the , but instead of creating your own dataset or using an existing external one, you'll export a trace dataset from Phoenix and log the evaluation results to Phoenix.

Install dependencies & Set environment variables

Connect to Phoenix

Note: if you're self-hosting Phoenix, swap your collector endpoint variable in the snippet below, and remove the Phoenix Client Headers variable.

Now that we have Phoenix configured, we can register that instance with OpenTelemetry, which will allow us to collect traces from our application here.

Prepare trace dataset

For the sake of making this guide fully runnable, we'll briefly generate some traces and track them in Phoenix. Typically, you would have already captured traces in Phoenix and would skip to "Download trace dataset from Phoenix"

Download trace dataset from Phoenix

Generate evaluations

Now that we have our trace dataset, we can generate evaluations for each trace. Evaluations can be generated in many different ways. Ultimately, we want to end up with a set of labels and/or scores for our traces.

You can generate evaluations using:

• Plain code

• Phoenix's

• Your own

• Other evaluation packages

As long as you format your evaluation results properly, you can upload them to Phoenix and visualize them in the UI.

Let's start with a simple example of generating evaluations using plain code. OpenAI has a habit of repeating jokes, so we'll generate evaluations to label whether a joke is a repeat of a previous joke.

We now have a DataFrame with a column for whether each joke is a repeat of a previous joke. Let's upload this to Phoenix.

Upload evaluations to Phoenix

Our evals_df has a column for the span_id and a column for the evaluation result. The span_id is what allows us to connect the evaluation to the correct trace in Phoenix. Phoenix will also automatically look for columns named "label" and "score" to display in the UI.

You should now see evaluations in the Phoenix UI!

From here you can continue collecting and evaluating traces, or move on to one of these other guides:

• If you're interested in more complex evaluation and evaluators, start with

• If you're ready to start testing your application in a more rigorous manner, check out

When To Use Toxicity Eval Template

The following shows the results of the toxicity Eval on a toxic dataset test to identify if the AI response is racist, biased, or toxic. The template variables are:

• text: the text to be classified

Toxicity Eval Template

How To Run the Toxicity Eval

Benchmark Results

This benchmark was obtained using notebook below. It was run using the as a ground truth dataset. Each example in the dataset was evaluating using the TOXICITY_PROMPT_TEMPLATE above, then the resulting labels were compared against the ground truth label in the benchmark dataset to generate the confusion matrices below.

GPT-4 Results

Note: Palm is not useful for Toxicity detection as it always returns "" string for toxic inputs

The following are simple functions on top of the LLM evals building blocks that are pre-tested with benchmark data.

Supported Models.

The models are instantiated and usable in the LLM Eval function. The models are also directly callable with strings.

We currently support a growing set of models for LLM Evals, please check out the .

From CSV

When manually creating a dataset (let's say collecting hypothetical questions and answers), the easiest way to start is by using a spreadsheet. Once you've collected the information, you can simply upload the CSV of your data to the Phoenix platform using the UI. You can also programmatically upload tabular data using Pandas as

From Pandas

From Objects

Sometimes you just want to upload datasets using plain objects as CSVs and DataFrames can be too restrictive about the keys.

Synthetic Data

One of the quickest ways of getting started is to produce synthetic queries using an LLM.

From Spans

If you have an application that is traced using instrumentation, you can quickly add any span or group of spans using the Phoenix UI.

To add a single span to a dataset, simply select the span in the trace details view. You should see an add to dataset button on the top right. From there you can select the dataset you would like to add it to and make any changes you might need to make before saving the example.

You can also use the filters on the spans table and select multiple spans to add to a specific dataset.

Prompts in Phoenix are versioned in a linear history, creating a comprehensive audit trail of all modifications. Each change is tracked, allowing you to:

• Review the complete history of a prompt

• Understand who made specific changes

• Revert to previous versions if needed

Creating a Tag

When you are ready to deploy a prompt to a certain environment (let's say staging), the best thing to do is to tag a specific version of your prompt as ready. By default Phoenix offers 3 tags, production, staging, and development but you can create your own tags as well.

Each tag can include an optional description to provide additional context about its purpose or significance. Tags are unique per prompt, meaning you cannot have two tags with the same name for the same prompt.

Creating a custom tag

It can be helpful to have custom tags to track different versions of a prompt. For example if you wanted to tag a certain prompt as the one that was used in your v0 release, you can create a custom tag with that name to keep track!

When creating a custom tag, you can provide:

• A name for the tag (must be a valid identifier)

• An optional description to provide context about the tag's purpose

Pulling a prompt by tag

Once a prompt version is tagged, you can pull this version of the prompt into any environment that you would like (an application, an experiment). Similar to git tags, prompt version tags let you create a "release" of a prompt (e.x. pushing a prompt to staging).

You can retrieve a prompt version by:

• Using the tag name directly (e.g., "production", "staging", "development")

• Using a custom tag name

• Using the latest version (which will return the most recent version regardless of tags)

For full details on how to use prompts in code, see

Listing tags

You can list all tags associated with a specific prompt version. The list is paginated, allowing you to efficiently browse through large numbers of tags. Each tag in the list includes:

• The tag's unique identifier

• The tag's name

• The tag's description (if provided)

This is particularly useful when you need to:

• Review all tags associated with a prompt version

• Verify which version is currently tagged for a specific environment

• Track the history of tag changes for a prompt version

Using the Client

Tag Naming Rules

Tag names must be valid identifiers: lowercase letters, numbers, hyphens, and underscores, starting and ending with a letter or number.

Examples: staging, production-v1, release-2024

Creating and Managing Tags

When To Use Hallucination Eval Template

This LLM Eval detects if the output of a model is a hallucination based on contextual data.

This Eval is specifically designed to detect hallucinations in generated answers from private or retrieved data. The Eval detects if an AI answer to a question is a hallucination based on the reference data used to generate the answer.

Hallucination Eval Template

How To Run the Hallucination Eval

The above Eval shows how to the the hallucination template for Eval detection.

Benchmark Results

This benchmark was obtained using notebook below. It was run using the as a ground truth dataset. Each example in the dataset was evaluating using the HALLUCINATION_PROMPT_TEMPLATE above, then the resulting labels were compared against the is_hallucination label in the HaluEval dataset to generate the confusion matrices below.

GPT-4 Results

This quickstart guide will show you through the basics of evaluating data from your LLM application.

Install Phoenix Evals

Prepare your dataset

The first thing you'll need is a dataset to evaluate. This could be your own collected or generated set of examples, or data you've exported from Phoenix traces. If you've already collected some trace data, this makes a great starting point.

For the sake of this guide however, we'll download some pre-existing data to evaluate. Feel free to substitute this with your own data, just be sure it includes the following columns:

• reference

• query

• response

Evaluate and Log Results

Set up evaluators (in this case for hallucinations and Q&A correctness), run the evaluations, and log the results to visualize them in Phoenix. We'll use OpenAI as our evaluation model for this example, but Phoenix also supports a number of other models. First, we need to add our OpenAI API key to our environment.

Explanation of the parameters used in run_evals above:

• dataframe - a pandas dataframe that includes the data you want to evaluate. This could be spans exported from Phoenix, or data you've brought in from elsewhere. This dataframe must include the columns expected by the evaluators you are using. To see the columns expected by each built-in evaluator, check the corresponding page in the Using Phoenix Evaluators section.

• evaluators - a list of built-in Phoenix evaluators to use.

• provide_explanation - a binary flag that instructs the evaluators to generate explanations for their choices.

Analyze Your Evaluations

Combine your evaluation results and explanations with your original dataset:

(Optional) Log Results to Phoenix

Note: You'll only be able to log evaluations to the Phoenix UI if you used a trace or span dataset exported from Phoenix as your dataset in this quickstart. If you've used your own outside dataset, you won't be able to log these results to Phoenix.

Provided you started from a trace dataset, you can log your evaluation results to Phoenix using .

The Agent Function Call eval can be used to determine how well a model selects a tool to use, extracts the right parameters from the user query, and generates the tool call code.

Function Calling Eval Template

Running an Agent Eval using the Function Calling Template

Parameters:

• df - a dataframe of cases to evaluate. The dataframe must have these columns to match the default template:

  • question - the query made to the model. If you've to evaluate, this will the llm.input_messages column in your exported data.

  • tool_call - information on the tool called and parameters included. If you've to evaluate, this will be the llm.function_call column in your exported data.

Parameter Extraction Only

This template instead evaluates only the parameter extraction step of a router:

Installation

First, install the Phoenix client library:

npm install @arizeai/phoenix-client

Creating a Prompt

Let's start by creating a simple prompt in Phoenix using the TypeScript client:

import { createClient } from "@arizeai/phoenix-client";
import { createPrompt, promptVersion } from "@arizeai/phoenix-client/prompts";

// Create a Phoenix client 
// (optional, the createPrompt function will create one if not provided)
const client = createClient({
  options: {
    baseUrl: "http://localhost:6006", // Change to your Phoenix server URL
    // If your Phoenix instance requires authentication:
    // headers: {
    //   Authorization: "bearer YOUR_API_KEY",
    // }
  }
});

// Define a simple summarization prompt
const summarizationPrompt = await createPrompt({
  client,
  name: "article-summarizer",
  description: "Summarizes an article into concise bullet points",
  version: promptVersion({
    description: "Initial version",
    templateFormat: "MUSTACHE",
    modelProvider: "OPENAI", // Could also be ANTHROPIC, GEMINI, etc.
    modelName: "gpt-3.5-turbo",
    template: [
      {
        role: "system",
        content: "You are an expert summarizer. Create clear, concise bullet points highlighting the key information."
      },
      {
        role: "user",
        content: "Please summarize the following {{topic}} article:\n\n{{article}}"
      }
    ],
  })
});

console.dir(summarizationPrompt);

Getting a Prompt

You can retrieve prompts by name, ID, version, or tag:

import { getPrompt } from "@arizeai/phoenix-client/prompts";

// Get by name (latest version)
const latestPrompt = await getPrompt({
  prompt: {
    name: "article-summarizer",
  }
});

// Get by specific version ID
const specificVersionPrompt = await getPrompt({ 
  prompt: {
    versionId: "abcd1234",
  },
});

// Get by tag (e.g., "production", "staging", "development")
const productionPrompt = await getPrompt({ 
  prompt: {
    name: "article-summarizer", 
    tag: "production", 
  }
});

Using a Prompt with SDKs

Phoenix makes it easy to use your prompts with various SDKs, no proprietary SDK necessary! Here's how to use a prompt with OpenAI:

import { getPrompt, toSDK } from "@arizeai/phoenix-client/prompts";
import OpenAI from "openai";

// Initialize your OpenAI client
const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

// Get your prompt
const prompt = await getPrompt({
  prompt: {
    name: "article-summarizer",
  },
});

// Make sure the prompt was properly fetched
if (!prompt) {
  throw new Error("Prompt not found");
}

// Transform the prompt to OpenAI format with variable values
const openaiParameters = toSDK({
  sdk: "openai", 
  prompt,
  variables: {
    topic: "technology",
    article:
      "Artificial intelligence has seen rapid advancement in recent years. Large language models like GPT-4 can now generate human-like text, code, and even create images from descriptions. This technology is being integrated into many industries, from healthcare to finance, transforming how businesses operate and people work.",
  },
});

// Make sure the prompt was successfully converted to parameters
if (!openaiParameters) {
  throw new Error("OpenAI parameters not found");
}

// Use the transformed parameters with OpenAI
const response = await openai.chat.completions.create({
  ...openaiParameters,
  // You can override any parameters here
  model: "gpt-4o-mini", // Override the model if needed
  stream: false,
});


console.log("Summary:", response.choices[0].message.content);

The Phoenix client natively supports passing your prompts to OpenAI, Anthropic, and the Vercel AI SDK.

Next Steps

• Check out the How to: Prompts section for details on how to test prompt changes

• Take a look a the TypeScript examples in the Phoenix client (https://github.com/Arize-ai/phoenix/tree/main/js/packages/phoenix-client/examples)

• Try out some Deno notebooks to experiment with prompts (https://github.com/Arize-ai/phoenix/tree/main/js/examples/notebooks)

LLM Evaluators

We provide LLM evaluators out of the box. These evaluators are vendor agnostic and can be instantiated with a Phoenix model wrapper:

from phoenix.experiments.evaluators import HelpfulnessEvaluator
from phoenix.evals.models import OpenAIModel

helpfulness_evaluator = HelpfulnessEvaluator(model=OpenAIModel())

Code Evaluators

Code evaluators are functions that evaluate the output of your LLM task that don't use another LLM as a judge. An example might be checking for whether or not a given output contains a link - which can be implemented as a RegEx match.

phoenix.experiments.evaluators contains some pre-built code evaluators that can be passed to the evaluators parameter in experiments.

from phoenix.experiments import run_experiment, MatchesRegex

# This defines a code evaluator for links
contains_link = MatchesRegex(
    pattern=r"[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&//=]*)",
    name="contains_link"
)

For a full list of code evaluators, please consult repo or API documentation.

Custom Evaluators

The simplest way to create an evaluator is just to write a Python function. By default, a function of one argument will be passed the output of an experiment run. These custom evaluators can either return a boolean or numeric value which will be recorded as the evaluation score.

def in_bounds(x):
    return 1 <= x <= 100

More complex evaluations can use additional information. These values can be accessed by defining a function with specific parameter names which are bound to special values:

These parameters can be used in any combination and any order to write custom complex evaluators!

pip install editdistance

def edit_distance(output, expected) -> int:
    return editdistance.eval(
        json.dumps(output, sort_keys=True), json.dumps(expected, sort_keys=True)
    )

For even more customization, use the create_evaluator decorator to further customize how your evaluations show up in the Experiments UI.

from phoenix.experiments.evaluators import create_evaluator

# the decorator can be used to set display properties
# `name` corresponds to the metric name shown in the UI
# `kind` indicates if the eval was made with a "CODE" or "LLM" evaluator
@create_evaluator(name="shorter?", kind="CODE")
def wordiness_evaluator(expected, output):
    reference_length = len(expected.split())
    output_length = len(output.split())
    return output_length < reference_length

Multiple Evaluators on Experiment Runs

Phoenix supports running multiple evals on a single experiment, allowing you to comprehensively assess your model's performance from different angles. When you provide multiple evaluators, Phoenix creates evaluation runs for every combination of experiment runs and evaluators.

from phoenix.experiments import run_experiment
from phoenix.experiments.evaluators import ContainsKeyword, MatchesRegex

experiment = run_experiment(
    dataset,
    task,
    evaluators=[
        ContainsKeyword("hello"),
        MatchesRegex(r"\d+"),
        custom_evaluator_function
    ]
)