Eval Library integrations - where Phoenix traces can be evaluated using an outside eval library, instead of Phoenix's eval library, and visualized in Phoenix.
Instrument LLM calls to AWS Bedrock via the boto3 client using the BedrockInstrumentor
boto3 provides Python bindings to AWS services, including Bedrock, which provides access to a number of foundation models. Calls to these models can be instrumented using OpenInference, enabling OpenTelemetry-compliant observability of applications built using these models. Traces collected using OpenInference can be viewed in Phoenix.
OpenInference Traces collect telemetry data about the execution of your LLM application. Consider using this instrumentation to understand how a Bedrock-managed models are being called inside a complex system and to troubleshoot issues such as extraction and response synthesis.
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see Terminal Setup.
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Connect to your Phoenix instance using the register function.
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
After connecting to your Phoenix server, instrument boto3 prior to initializing a bedrock-runtime client. All clients created after instrumentation will send traces on all calls to invoke_model.
class BedrockModel:
model_id: str = "anthropic.claude-v2"
"""The model name to use."""
temperature: float = 0.0
"""What sampling temperature to use."""
max_tokens: int = 256
"""The maximum number of tokens to generate in the completion."""
top_p: float = 1
"""Total probability mass of tokens to consider at each step."""
top_k: int = 256
"""The cutoff where the model no longer selects the words"""
stop_sequences: List[str] = field(default_factory=list)
"""If the model encounters a stop sequence, it stops generating further tokens. """
session: Any = None
"""A bedrock session. If provided, a new bedrock client will be created using this session."""
client = None
"""The bedrock session client. If unset, a new one is created with boto3."""
max_content_size: Optional[int] = None
"""If you're using a fine-tuned model, set this to the maximum content size"""
extra_parameters: Dict[str, Any] = field(default_factory=dict)
"""Any extra parameters to add to the request body (e.g., countPenalty for a21 models)"""
To Authenticate, the following code is used to instantiate a session and the session is used with Phoenix Evals
import boto3
# Create a Boto3 session
session = boto3.session.Session(
aws_access_key_id='ACCESS_KEY',
aws_secret_access_key='SECRET_KEY',
region_name='us-east-1' # change to your preferred AWS region
)
#If you need to assume a role
# Creating an STS client
sts_client = session.client('sts')
# (optional - if needed) Assuming a role
response = sts_client.assume_role(
RoleArn="arn:aws:iam::......",
RoleSessionName="AssumeRoleSession1",
#(optional) if MFA Required
SerialNumber='arn:aws:iam::...',
#Insert current token, needs to be run within x seconds of generation
TokenCode='PERIODIC_TOKEN'
)
# Your temporary credentials will be available in the response dictionary
temporary_credentials = response['Credentials']
# Creating a new Boto3 session with the temporary credentials
assumed_role_session = boto3.Session(
aws_access_key_id=temporary_credentials['AccessKeyId'],
aws_secret_access_key=temporary_credentials['SecretAccessKey'],
aws_session_token=temporary_credentials['SessionToken'],
region_name='us-east-1'
)
client_bedrock = assumed_role_session.client("bedrock-runtime")
# Arize Model Object - Bedrock ClaudV2 by default
model = BedrockModel(client=client_bedrock)
model("Hello there, how are you?")
# Output: "As an artificial intelligence, I don't have feelings,
# but I'm here and ready to assist you. How can I help you today?"
Amazon Bedrock Agents Tracing
Instrument LLM calls to AWS Bedrock via the boto3 client using the BedrockInstrumentor
Amazon Bedrock Agents allow you to easily define, deploy, and manage agents on your AWS infrastructure. Traces on invocations of these agents can be captured using OpenInference and viewed in Phoenix.
This instrumentation will capture data on LLM calls, action group invocations (as tools), knowledgebase lookups, and more.
Launch Phoenix
Install
pip install openinference-instrumentation-bedrock
Setup
Connect to your Phoenix instance using the register function.
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
After connecting to your Phoenix server, instrument boto3 prior to initializing a bedrock-runtime client. All clients created after instrumentation will send traces on all calls to invoke_model, invoke_agent, and their streaming variations.
Use the register function to connect your application to Phoenix:
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run Anthropic
A simple Anthropic application that is now instrumented
class AnthropicModel(BaseModel):
model: str = "claude-2.1"
"""The model name to use."""
temperature: float = 0.0
"""What sampling temperature to use."""
max_tokens: int = 256
"""The maximum number of tokens to generate in the completion."""
top_p: float = 1
"""Total probability mass of tokens to consider at each step."""
top_k: int = 256
"""The cutoff where the model no longer selects the words."""
stop_sequences: List[str] = field(default_factory=list)
"""If the model encounters a stop sequence, it stops generating further tokens."""
extra_parameters: Dict[str, Any] = field(default_factory=dict)
"""Any extra parameters to add to the request body (e.g., countPenalty for a21 models)"""
max_content_size: Optional[int] = None
"""If you're using a fine-tuned model, set this to the maximum content size"""
Usage
In this section, we will showcase the methods and properties that our EvalModels have. First, instantiate your model from the. Once you've instantiated your model, you can get responses from the LLM by simply calling the model and passing a text string.
model = #Instantiate your Anthropic model here
model("Hello there, how are you?")
# Output: "As an artificial intelligence, I don't have feelings,
# but I'm here and ready to assist you. How can I help you today?"
Google Gen AI
Google GenAI is a suite of AI tools and models from Google Cloud, designed to help businesses build, deploy, and scale AI applications.
Set the GEMINI_API_KEY environment variable. To use the GenAI SDK with Vertex AI instead of the Developer API, refer to Google's guide on setting the required environment variables.
export GEMINI_API_KEY=[your_key_here]
Use the register function to connect your application to Phoenix.
from phoenix.otel import register
# Configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Observe
Now that you have tracing setup, all GenAI SDK requests will be streamed to Phoenix for observability and evaluation.
import os
from google import genai
def send_message_multi_turn() -> tuple[str, str]:
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
chat = client.chats.create(model="gemini-2.0-flash-001")
response1 = chat.send_message("What is the capital of France?")
response2 = chat.send_message("Why is the sky blue?")
return response1.text or "", response2.text or ""
This instrumentation will support tool calling soon. Refer to this page for the status.
Gemini Evals
Configure and run Gemini for evals
GeminiModel
class GeminiModel:
project: Optional[str] = None
location: Optional[str] = None
credentials: Optional["Credentials"] = None
model: str = "gemini-pro"
default_concurrency: int = 5
temperature: float = 0.0
max_tokens: int = 256
top_p: float = 1
top_k: int = 32
To authenticate with Gemini, you must pass either your credentials or a project, location pair. In the following example, we quickly instantiate the Gemini model as follows:
project = "my-project-id"
location = "us-central1" # as an example
model = GeminiModel(project=project, location=location)
model("Hello there, this is a tesst if you are working?")
# Output: "Hello world, I am working!"
LiteLLM
LiteLLM is an open-source platform that provides a unified interface to manage and access over 100 LLMs from various providers.
LiteLLM allows developers to call all LLM APIs using the openAI format. LiteLLM Proxy is a proxy server to call 100+ LLMs in OpenAI format. Both are supported by this auto-instrumentation.
Any calls made to the following functions will be automatically captured by this integration:
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see Terminal Setup.
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Use the register function to connect your application to Phoenix:
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Add any API keys needed by the models you are using with LiteLLM.
import os
os.environ["OPENAI_API_KEY"] = "PASTE_YOUR_API_KEY_HERE"
Run LiteLLM
You can now use LiteLLM as normal and calls will be traces in Phoenix.
import litellm
completion_response = litellm.completion(model="gpt-3.5-turbo",
messages=[{"content": "What's the capital of China?", "role": "user"}])
print(completion_response)
Need to install the extra dependency litellm>=1.0.3
class LiteLLMModel(BaseEvalModel):
model: str = "gpt-3.5-turbo"
"""The model name to use."""
temperature: float = 0.0
"""What sampling temperature to use."""
max_tokens: int = 256
"""The maximum number of tokens to generate in the completion."""
top_p: float = 1
"""Total probability mass of tokens to consider at each step."""
num_retries: int = 6
"""Maximum number to retry a model if an RateLimitError, OpenAIError, or
ServiceUnavailableError occurs."""
request_timeout: int = 60
"""Maximum number of seconds to wait when retrying."""
model_kwargs: Dict[str, Any] = field(default_factory=dict)
"""Model specific params"""
You can choose among multiple models supported by LiteLLM. Make sure you have set the right environment variables set prior to initializing the model. For additional information about the environment variables for specific model providers visit: LiteLLM provider specific params
Here is an example of how to initialize LiteLLMModel for llama3 using ollama.
import os
from phoenix.evals import LiteLLMModel
os.environ["OLLAMA_API_BASE"] = "http://localhost:11434"
model = LiteLLMModel(model="ollama/llama3")
MistralAI
Mistral AI develops open-weight large language models, focusing on efficiency, customization, and cost-effective AI solutions.
Set the MISTRAL_API_KEY environment variable to authenticate calls made using the SDK.
export MISTRAL_API_KEY=[your_key_here]
Connect to your Phoenix instance using the register function.
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run Mistral
import os
from mistralai import Mistral
from mistralai.models import UserMessage
api_key = os.environ["MISTRAL_API_KEY"]
model = "mistral-tiny"
client = Mistral(api_key=api_key)
chat_response = client.chat.complete(
model=model,
messages=[UserMessage(content="What is the best French cheese?")],
)
print(chat_response.choices[0].message.content)
Observe
Now that you have tracing setup, all invocations of Mistral (completions, chat completions, embeddings) will be streamed to your running Phoenix for observability and evaluation.
# model = Instantiate your MistralAIModel here
model("Hello there, how are you?")
# Output: "As an artificial intelligence, I don't have feelings,
# but I'm here and ready to assist you. How can I help you today?"
Groq
Groq provides ultra-low latency inference for LLMs through its custom-built LPU™ architecture.
Groq provides low latency and lightning-fast inference for AI models. Arize supports instrumenting Groq API calls, including role types such as system, user, and assistant messages, as well as tool use. You can create a free GroqCloud account and generate a Groq API Key here to get started.
Connect to your Phoenix instance using the register function.
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run Groq
A simple Groq application that is now instrumented
import os
from groq import Groq
client = Groq(
# This is the default and can be omitted
api_key=os.environ.get("GROQ_API_KEY"),
)
chat_completion = client.chat.completions.create(
messages=[
{
"role": "user",
"content": "Explain the importance of low latency LLMs",
}
],
model="mixtral-8x7b-32768",
)
print(chat_completion.choices[0].message.content)
Observe
Now that you have tracing setup, all invocations of pipelines will be streamed to your running Phoenix for observability and evaluation.
Add your OpenAI API key as an environment variable:
export OPENAI_API_KEY=[your_key_here]
Use the register function to connect your application to Phoenix:
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed dependencies
)
Now that you have tracing setup, all invocations of OpenAI (completions, chat completions, embeddings) will be streamed to your running Phoenix for observability and evaluation.
Need to install the extra dependencies openai>=1.0.0
class OpenAIModel:
api_key: Optional[str] = field(repr=False, default=None)
"""Your OpenAI key. If not provided, will be read from the environment variable"""
organization: Optional[str] = field(repr=False, default=None)
"""
The organization to use for the OpenAI API. If not provided, will default
to what's configured in OpenAI
"""
base_url: Optional[str] = field(repr=False, default=None)
"""
An optional base URL to use for the OpenAI API. If not provided, will default
to what's configured in OpenAI
"""
model: str = "gpt-4"
"""Model name to use. In of azure, this is the deployment name such as gpt-35-instant"""
temperature: float = 0.0
"""What sampling temperature to use."""
max_tokens: int = 256
"""The maximum number of tokens to generate in the completion.
-1 returns as many tokens as possible given the prompt and
the models maximal context size."""
top_p: float = 1
"""Total probability mass of tokens to consider at each step."""
frequency_penalty: float = 0
"""Penalizes repeated tokens according to frequency."""
presence_penalty: float = 0
"""Penalizes repeated tokens."""
n: int = 1
"""How many completions to generate for each prompt."""
model_kwargs: Dict[str, Any] = field(default_factory=dict)
"""Holds any model parameters valid for `create` call not explicitly specified."""
batch_size: int = 20
"""Batch size to use when passing multiple documents to generate."""
request_timeout: Optional[Union[float, Tuple[float, float]]] = None
"""Timeout for requests to OpenAI completion API. Default is 600 seconds."""
All models newer than GPT 3.5 Turbo are tested regularly. If you're using an older model than that, you may run into deprecated API parameters.
To authenticate with OpenAI you will need, at a minimum, an API key. The model class will look for it in your environment, or you can pass it via argument as shown above. In addition, you can choose the specific name of the model you want to use and its configuration parameters. The default values specified above are common default values from OpenAI. Quickly instantiate your model as follows:
model = OpenAI()
model("Hello there, this is a test if you are working?")
# Output: "Hello! I'm working perfectly. How can I assist you today?"
Azure OpenAI
The code snippet below shows how to initialize OpenAIModel for Azure:
model = OpenAIModel(
model="gpt-35-turbo-16k",
azure_endpoint="https://arize-internal-llm.openai.azure.com/",
api_version="2023-09-15-preview",
)
Note that the model param is actually the engine of your deployment. You may get a DeploymentNotFound error if this parameter is not correct. You can find your engine param in the Azure OpenAI playground.
\
Azure OpenAI supports specific options:
api_version: str = field(default=None)
"""
The verion of the API that is provisioned
https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#rest-api-versioning
"""
azure_endpoint: Optional[str] = field(default=None)
"""
The endpoint to use for azure openai. Available in the azure portal.
https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal#create-a-resource
"""
azure_deployment: Optional[str] = field(default=None)
azure_ad_token: Optional[str] = field(default=None)
azure_ad_token_provider: Optional[Callable[[], str]] = field(default=None)
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see Terminal Setup.
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Add your OpenAI API key as an environment variable:
export OPENAI_API_KEY=[your_key_here]
Use the register function to connect your application to Phoenix:
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="agents", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed dependencies
)
Run your agentscode.
from agents import Agent, Runner
agent = Agent(name="Assistant", instructions="You are a helpful assistant")
result = Runner.run_sync(agent, "Write a haiku about recursion in programming.")
print(result.final_output)
See Google's guide on setting up your environment for the Google Cloud AI Platform. You can also store your Project ID in the CLOUD_ML_PROJECT_ID environment variable.
Use the register function to connect your application to Phoenix:
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run VertexAI
import vertexai
from vertexai.generative_models import GenerativeModel
vertexai.init(location="us-central1")
model = GenerativeModel("gemini-1.5-flash")
print(model.generate_content("Why is sky blue?").text)
Observe
Now that you have tracing setup, all invocations of Vertex models will be streamed to your running Phoenix for observability and evaluation.
To authenticate with VertexAI, you must pass either your credentials or a project, location pair. In the following example, we quickly instantiate the VertexAI model as follows:
project = "my-project-id"
location = "us-central1" # as an example
model = VertexAIModel(project=project, location=location)
model("Hello there, this is a tesst if you are working?")
# Output: "Hello world, I am working!"
Frameworks
Agno
Agno is an open-source Python framework for building lightweight, model-agnostic AI agents with built-in memory, knowledge, tools, and reasoning capabilities
Phoenix provides seamless observability and tracing for Agno agents through the OpenInference instrumentation package. This integration automatically captures agent interactions, tool usage, reasoning steps, and multi-agent conversations, giving you complete visibility into your Agno applications. Monitor performance, debug issues, and evaluate agent behavior in real-time as your agents execute complex workflows and collaborate in teams.
Agno is a lightweight, high-performance Python framework for building AI agents with tools, memory, and reasoning capabilities. It enables developers to create autonomous agents that can perform complex tasks, access knowledge bases, and collaborate in multi-agent teams. With support for 23+ model providers and lightning-fast performance (~3μs instantiation), Agno is designed for production-ready AI applications.
Agno Traces in Phoenix
Key Features
Model Agnostic: Connect to OpenAI, Anthropic, Google, and 20+ other providers
Lightning Fast: Agents instantiate in ~3μs with minimal memory footprint
Built-in Reasoning: First-class support for chain-of-thought and reasoning models
Multi-Modal: Native support for text, image, audio, and video processing
Agentic RAG: Advanced retrieval-augmented generation with hybrid search
Multi-Agent Teams: Coordinate multiple agents for complex workflows
Production Ready: Pre-built FastAPI routes and monitoring capabilities
Use the register function to connect your application to Phoenix:
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run Agno
from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.tools.duckduckgo import DuckDuckGoTools
agent = Agent(
model=OpenAIChat(id="gpt-4o-mini"),
tools=[DuckDuckGoTools()],
markdown=True,
debug_mode=True,
)
agent.run("What is currently trending on Twitter?")
Observe
Now that you have tracing setup, all invocations of Agno agents will be streamed to Phoenix for observability and evaluation.
AutoGen is an open-source Python framework for orchestrating multi-agent LLM interactions with shared memory and tool integrations to build scalable AI workflows
AutoGen is an agent framework from Microsoft that allows for complex Agent creation. It is unique in its ability to create multiple agents that work together.
The AutoGen Agent framework allows creation of multiple agents and connection of those agents to work together to accomplish tasks.
import os
# Add Phoenix API Key for tracing
PHOENIX_API_KEY = "ADD YOUR API KEY"
os.environ["PHOENIX_CLIENT_HEADERS"] = f"api_key={PHOENIX_API_KEY}"
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pip install arize-phoenix
phoenix serve
For details on customizing a local terminal deployment, see Terminal Setup.
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Install
Phoenix instruments Autogen by instrumenting the underlying model library it's using. If your agents are set up to call OpenAI, use our OpenAI instrumentor per the example below.
If your agents are using a different model, be sure to instrument that model instead by installing its respective OpenInference library.
Connect to your Phoenix instance using the register function.
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run Autogen
From here you can use Autogen as normal, and Phoenix will automatically trace any model calls made.
Observe
The Phoenix support is simple in its first incarnation but allows for capturing all of the prompt and responses that occur under the framework between each agent.
The individual prompt and responses are captured directly through OpenAI calls. If you're using a different underlying model provider than OpenAI, instrument your application using the respective instrumentor instead.
BeeAI is an open-source platform that enables developers to discover, run, and compose AI agents from any framework, facilitating the creation of interoperable multi-agent systems
This module provides automatic instrumentation for BeeAI framework. It integrates seamlessly with the @opentelemetry/sdk-trace-node package to collect and export telemetry data.
CrewAI is an open-source Python framework for orchestrating role-playing, autonomous AI agents into collaborative “crews” and “flows,” combining high-level simplicity with fine-grained control.
import os
# Add Phoenix API Key for tracing
PHOENIX_API_KEY = "ADD YOUR API KEY"
os.environ["PHOENIX_CLIENT_HEADERS"] = f"api_key={PHOENIX_API_KEY}"
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "https://app.phoenix.arize.com"
Launch your local Phoenix instance:
pip install arize-phoenix
phoenix serve
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
See Terminal for more details
docker pull arizephoenix/phoenix:latest
Run your containerized instance:
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
CrewAI uses either Langchain or LiteLLM under the hood to call models, depending on the version.
If you're using CrewAI<0.63.0, we recommend installing our openinference-instrumentation-langchain library to get visibility of LLM calls.
If you're using CrewAI>= 0.63.0, we recommend instead adding our openinference-instrumentation-litellm library to get visibility of LLM calls.
Setup
Connect to your Phoenix instance using the register function.
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run CrewAI
From here, you can run CrewAI as normal
import os
from crewai import Agent, Task, Crew, Process
from crewai_tools import SerperDevTool
os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"
os.environ["SERPER_API_KEY"] = "YOUR_SERPER_API_KEY"
search_tool = SerperDevTool()
# Define your agents with roles and goals
researcher = Agent(
role='Senior Research Analyst',
goal='Uncover cutting-edge developments in AI and data science',
backstory="""You work at a leading tech think tank.
Your expertise lies in identifying emerging trends.
You have a knack for dissecting complex data and presenting actionable insights.""",
verbose=True,
allow_delegation=False,
# You can pass an optional llm attribute specifying what model you wanna use.
# llm=ChatOpenAI(model_name="gpt-3.5", temperature=0.7),
tools=[search_tool]
)
writer = Agent(
role='Tech Content Strategist',
goal='Craft compelling content on tech advancements',
backstory="""You are a renowned Content Strategist, known for your insightful and engaging articles.
You transform complex concepts into compelling narratives.""",
verbose=True,
allow_delegation=True
)
# Create tasks for your agents
task1 = Task(
description="""Conduct a comprehensive analysis of the latest advancements in AI in 2024.
Identify key trends, breakthrough technologies, and potential industry impacts.""",
expected_output="Full analysis report in bullet points",
agent=researcher
)
task2 = Task(
description="""Using the insights provided, develop an engaging blog
post that highlights the most significant AI advancements.
Your post should be informative yet accessible, catering to a tech-savvy audience.
Make it sound cool, avoid complex words so it doesn't sound like AI.""",
expected_output="Full blog post of at least 4 paragraphs",
agent=writer
)
# Instantiate your crew with a sequential process
crew = Crew(
agents=[researcher, writer],
tasks=[task1, task2],
verbose=2, # You can set it to 1 or 2 to different logging levels
process = Process.sequential
)
# Get your crew to work!
result = crew.kickoff()
print("######################")
print(result)
Observe
Now that you have tracing setup, all calls to your Crew will be streamed to your running Phoenix for observability and evaluation.
Resources
DSPy
DSPy is an open-source Python framework for declaratively programming modular LLM pipelines and automatically optimizing prompts and model weights
Instrument and observe your DSPy application via the DSPyInstrumentor
DSPy is a framework for automatically prompting and fine-tuning language models. It provides composable and declarative APIs that allow developers to describe the architecture of their LLM application in the form of a "module" (inspired by PyTorch's nn.Module). It them compiles these modules using "teleprompters" that optimize the module for a particular task. The term "teleprompter" is meant to evoke "prompting at a distance," and could involve selecting few-shot examples, generating prompts, or fine-tuning language models.
Phoenix makes your DSPy applications observable by visualizing the underlying structure of each call to your compiled DSPy module.
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see Terminal Setup.
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
DSPy uses LiteLLM under the hood to make some calls. By adding the OpenInference library for LiteLLM, you'll be able to see additional information like token counts on your traces.
Setup
Connect to your Phoenix instance using the register function.
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run DSPy
Now run invoke your compiled DSPy module. Your traces should appear inside of Phoenix.
class BasicQA(dspy.Signature):
"""Answer questions with short factoid answers."""
question = dspy.InputField()
answer = dspy.OutputField(desc="often between 1 and 5 words")
if __name__ == "__main__":
turbo = dspy.OpenAI(model="gpt-3.5-turbo")
dspy.settings.configure(lm=turbo)
with using_attributes(
session_id="my-test-session",
user_id="my-test-user",
metadata={
"test-int": 1,
"test-str": "string",
"test-list": [1, 2, 3],
"test-dict": {
"key-1": "val-1",
"key-2": "val-2",
},
},
tags=["tag-1", "tag-2"],
prompt_template_version="v1.0",
prompt_template_variables={
"city": "Johannesburg",
"date": "July 11th",
},
):
# Define the predictor.
generate_answer = dspy.Predict(BasicQA)
# Call the predictor on a particular input.
pred = generate_answer(
question="What is the capital of the united states?" # noqa: E501
) # noqa: E501
print(f"Predicted Answer: {pred.answer}")
Observe
Now that you have tracing setup, all predictions will be streamed to your running Phoenix for observability and evaluation.
Analyzing and troubleshooting what happens under the hood can be challenging without proper insights. By integrating your Flowise application with Phoenix, you can monitor traces and gain robust observability into your chatflows and agentflows.
Viewing Flowise traces in Phoenix
Access Configurations: Navigate to settings in your chatflow or agentflow and find configurations.
Connect to Phoenix: Go to the Analyze Chatflow tab and configure your application with Phoenix. Get your API key from your Phoenix instance to create your credentials. Be sure to name your project and confirm that the Phoenix toggle is enabled before saving.
Note: If you are using using an environment that is not Phoenix Cloud, you may need to modify the Endpoint field.
View Traces: In Phoenix, you will find your project under the Projects tab. Click into this to view and analyze traces as you test your application.
Store and Experiment: Optionally, you can also filter traces, store traces in a dataset to run experiments, analyze patterns, and optimize your workflows over time.
Guardrails is an open-source Python framework for adding programmable input/output validators to LLM applications, ensuring safe, structured, and compliant model interactions
import os
# Add Phoenix API Key for tracing
PHOENIX_API_KEY = "ADD YOUR API KEY"
os.environ["PHOENIX_CLIENT_HEADERS"] = f"api_key={PHOENIX_API_KEY}"
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pip install arize-phoenix
phoenix serve
For details on customizing a local terminal deployment, see Terminal Setup.
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Connect to your Phoenix instance using the register function.
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run Guardrails
From here, you can run Guardrails as normal:
from guardrails import Guard
from guardrails.hub import TwoWords
import openai
guard = Guard().use(
TwoWords(),
)
response = guard(
llm_api=openai.chat.completions.create,
prompt="What is another name for America?",
model="gpt-3.5-turbo",
max_tokens=1024,
)
print(response)
Observe
Now that you have tracing setup, all invocations of underlying models used by Guardrails (completions, chat completions, embeddings) will be streamed to your running Phoenix for observability and evaluation. Additionally, Guards will be present as a new span kind in Phoenix.
import os
# Add Phoenix API Key for tracing
PHOENIX_API_KEY = "ADD YOUR API KEY"
os.environ["PHOENIX_CLIENT_HEADERS"] = f"api_key={PHOENIX_API_KEY}"
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pip install arize-phoenix
phoenix serve
For details on customizing a local terminal deployment, see Terminal Setup.
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Use the register function to connect your application to Phoenix:
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run Haystack
Phoenix's auto-instrumentor collects any traces from Haystack Pipelines. If you are using Haystack but not using Pipelines, you won't see traces appearing in Phoenix automatically.
If you don't want to use Haystack pipelines but still want tracing in Phoenix, you can use instead of this auto-instrumentor.
From here, you can set up your Haystack app as normal:
from haystack import Pipeline
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders.prompt_builder import PromptBuilder
prompt_template = """
Answer the following question.
Question: {{question}}
Answer:
"""
# Initialize the pipeline
pipeline = Pipeline()
# Initialize the OpenAI generator component
llm = OpenAIGenerator(model="gpt-3.5-turbo")
prompt_builder = PromptBuilder(template=prompt_template)
# Add the generator component to the pipeline
pipeline.add_component("prompt_builder", prompt_builder)
pipeline.add_component("llm", llm)
pipeline.connect("prompt_builder", "llm")
# Define the question
question = "What is the location of the Hanging Gardens of Babylon?"
Observe
Now that you have tracing setup, all invocations of pipelines will be streamed to your running Phoenix for observability and evaluation.
Hugging Face smolagents is a minimalist Python library for building powerful AI agents with simple abstractions, tool integrations, and flexible LLM support
How to use the SmolagentsInstrumentor to trace smolagents by Hugging Face
smolagents is a minimalist AI agent framework developed by Hugging Face, designed to simplify the creation and deployment of powerful agents with just a few lines of code. It focuses on simplicity and efficiency, making it easy for developers to leverage large language models (LLMs) for various applications.
Phoenix provides auto-instrumentation, allowing you to track and visualize every step and call made by your agent.
Launch Phoenix
We have several code samples below on different ways to integrate with smolagents, based on how you want to use Phoenix.
import os
# Add Phoenix API Key for tracing
PHOENIX_API_KEY = "ADD YOUR API KEY"
os.environ["PHOENIX_CLIENT_HEADERS"] = f"api_key={PHOENIX_API_KEY}"
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pip install arize-phoenix
phoenix serve
For details on customizing a local terminal deployment, see Terminal Setup.
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Connect to your Phoenix instance using the register function.
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Create & Run an Agent
Create your Hugging Face Model, and at every run, traces will be sent to Phoenix.
from smolagents import (
CodeAgent,
ToolCallingAgent,
ManagedAgent,
DuckDuckGoSearchTool,
VisitWebpageTool,
HfApiModel,
)
model = HfApiModel()
agent = ToolCallingAgent(
tools=[DuckDuckGoSearchTool(), VisitWebpageTool()],
model=model,
)
managed_agent = ManagedAgent(
agent=agent,
name="managed_agent",
description="This is an agent that can do web search.",
)
manager_agent = CodeAgent(
tools=[],
model=model,
managed_agents=[managed_agent],
)
manager_agent.run(
"If the US keeps its 2024 growth rate, how many years will it take for the GDP to double?"
)
Observe
Now that you have tracing setup, all invocations and steps of your Agent will be streamed to your running Phoenix for observability and evaluation.
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see Terminal Setup.
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Be sure you also install the OpenInference library for the underlying model you're using along with Instructor. For example, if you're using OpenAI calls directly, you would also add: openinference-instrumentation-openai
Setup
Connect to your Phoenix instance using the register function.
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run Instructor
From here you can use instructor as normal.
import instructor
from pydantic import BaseModel
from openai import OpenAI
# Define your desired output structure
class UserInfo(BaseModel):
name: str
age: int
# Patch the OpenAI client
client = instructor.from_openai(OpenAI())
# Extract structured data from natural language
user_info = client.chat.completions.create(
model="gpt-3.5-turbo",
response_model=UserInfo,
messages=[{"role": "user", "content": "John Doe is 30 years old."}],
)
print(user_info.name)
#> John Doe
print(user_info.age)
#> 30
Observe
Now that you have tracing setup, all invocations of your underlying model (completions, chat completions, embeddings) and instructor triggers will be streamed to your running Phoenix for observability and evaluation.
LlamaIndex
LlamaIndex is an open-source framework that streamlines connecting, ingesting, indexing, and retrieving structured or unstructured data to power efficient, data-aware language model applications.
How to use the python LlamaIndexInstrumentor to trace LlamaIndex
LlamaIndex is a data framework for your LLM application. It's a powerful framework by which you can build an application that leverages RAG (retrieval-augmented generation) to super-charge an LLM with your own data. RAG is an extremely powerful LLM application model because it lets you harness the power of LLMs such as OpenAI's GPT but tuned to your data and use-case.
For LlamaIndex, tracing instrumentation is added via an OpenTelemetry instrumentor aptly named the LlamaIndexInstrumentor . This callback is what is used to create spans and send them to the Phoenix collector.
Launch Phoenix
Phoenix supports LlamaIndex's latest instrumentation paradigm. This paradigm requires LlamaIndex >= 0.10.43. For legacy support, see below.
Phoenix Developer Edition is another name for LlamaTrace
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint and API Key:
import os
# Add Phoenix API Key for tracing
PHOENIX_API_KEY = "ADD YOUR API KEY"
os.environ["PHOENIX_CLIENT_HEADERS"] = f"api_key={PHOENIX_API_KEY}"
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pip install arize-phoenix
phoenix serve
For details on customizing a local terminal deployment, see Terminal Setup.
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Initialize the LlamaIndexInstrumentor before your application code.
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
from phoenix.otel import register
tracer_provider = register()
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)
Run LlamaIndex
You can now use LlamaIndex as normal, and tracing will be automatically captured and sent to your Phoenix instance.
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
import os
os.environ["OPENAI_API_KEY"] = "YOUR OPENAI API KEY"
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("Some question about the data should go here")
print(response)
# Phoenix can display in real time the traces automatically
# collected from your LlamaIndex application.
import phoenix as px
# Look for a URL in the output to open the App in a browser.
px.launch_app()
# The App is initially empty, but as you proceed with the steps below,
# traces will appear automatically as your LlamaIndex application runs.
from llama_index.core import set_global_handler
set_global_handler("arize_phoenix")
# Run all of your LlamaIndex applications as usual and traces
# will be collected and displayed in Phoenix.
Legacy (<0.10.0)
If you are using an older version of llamaIndex (pre-0.10), you can still use phoenix. You will have to be using arize-phoenix>3.0.0 and downgrade openinference-instrumentation-llama-index<1.0.0
# Phoenix can display in real time the traces automatically
# collected from your LlamaIndex application.
import phoenix as px
# Look for a URL in the output to open the App in a browser.
px.launch_app()
# The App is initially empty, but as you proceed with the steps below,
# traces will appear automatically as your LlamaIndex application runs.
import llama_index
llama_index.set_global_handler("arize_phoenix")
# Run all of your LlamaIndex applications as usual and traces
# will be collected and displayed in Phoenix.
LlamaIndex Workflows Tracing
How to use the python LlamaIndexInstrumentor to trace LlamaIndex Workflows
LlamaIndex Workflows are a subset of the LlamaIndex package specifically designed to support agent development.
Our LlamaIndexInstrumentor automatically captures traces for LlamaIndex Workflows agents. If you've already enabled that instrumentor, you do not need to complete the steps below.
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see Terminal Setup.
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Initialize the LlamaIndexInstrumentor before your application code. This instrumentor will trace both LlamaIndex Workflows calls, as well as calls to the general LlamaIndex package.
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
from phoenix.otel import register
tracer_provider = register()
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)
Run LlamaIndex Workflows
By instrumenting LlamaIndex, spans will be created whenever an agent is invoked and will be sent to the Phoenix server for collection.
Observe
Now that you have tracing setup, all invocations of chains will be streamed to your running Phoenix for observability and evaluation.
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see Terminal Setup.
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Use the register function to connect your application to Phoenix:
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run LangChain
By instrumenting LangChain, spans will be created whenever a chain is run and will be sent to the Phoenix server for collection.
This module provides automatic instrumentation for LangChain.js, more specifically, the @langchain/core module. which may be used in conjunction with @opentelemetry/sdk-trace-node.
To load the LangChain instrumentation, manually instrument the @langchain/core/callbacks/manager module. The callbacks manager must be manually instrumented due to the non-traditional module structure in @langchain/core. Additional instrumentations can be registered as usual in the registerInstrumentations function.
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import {
LangChainInstrumentation
} from "@arizeai/openinference-instrumentation-langchain";
import * as CallbackManagerModule from "@langchain/core/callbacks/manager";
const provider = new NodeTracerProvider();
provider.register();
const lcInstrumentation = new LangChainInstrumentation();
// LangChain must be manually instrumented as it doesn't have
// a traditional module structure
lcInstrumentation.manuallyInstrument(CallbackManagerModule);
Support
Instrumentation version >1.0.0 supports both attribute masking and context attribute propagation to spans.
Phoenix has first-class support for LangGraph applications.
LangGraph is supported by our LangChain instrumentor. If you've already set up instrumentation with LangChain, you don't need to complete the set up below
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see Terminal Setup.
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Install the OpenInference Langchain library before your application code. Our LangChainInstrumentor works for both standard LangChain applications and for LangGraph agents.
Setup
Use the register function to connect your application to Phoenix:
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-llm-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run LangGraph
By instrumenting LangGraph, spans will be created whenever an agent is invoked and will be sent to the Phoenix server for collection.
Observe
Now that you have tracing setup, all invocations of chains will be streamed to your running Phoenix for observability and evaluation.
Langflow is an open-source visual framework that enables developers to rapidly design, prototype, and deploy custom applications powered by large language models (LLMs)
Note: This Langflow integration is for Phoenix Cloud
Start Docker Desktop
Start Docker Desktop, build the images, and run the container (this will take around 10 minutes the first time)
Go into your terminal into the Langflow directory and run the following commands
docker compose -f docker/dev.docker-compose.yml down || true
docker compose -f docker/dev.docker-compose.yml up --remove-orphans
AgentExecutor Trace is Arize Phoenix instrumentation to capture what's happening with the LangChain being ran during the Langflow components
The other UUID trace is the native Langflow tracing.
Mastra
Mastra is an open-source TypeScript AI agent framework designed for building production-ready AI applications with agents, workflows, RAG, and observability
Initialize OpenTelemetry tracing for your Mastra application:
import { Mastra } from "@mastra/core";
import {
OpenInferenceOTLPTraceExporter,
isOpenInferenceSpan,
} from "@arizeai/openinference-mastra";
export const mastra = new Mastra({
// ... other config
telemetry: {
serviceName: "openinference-mastra-agent", // you can rename this to whatever you want to appear in the Phoenix UI
enabled: true,
export: {
type: "custom",
exporter: new OpenInferenceOTLPTraceExporter({
url: process.env.PHOENIX_COLLECTOR_ENDPOINT,
headers: {
Authorization: `Bearer ${process.env.PHOENIX_API_KEY}`, // if you're self-hosting Phoenix without auth, you can remove this header
},
// optional: filter out http, and other node service specific spans
// they will still be exported to Mastra, but not to the target of
// this exporter
spanFilter: isOpenInferenceSpan,
}),
},
},
});
From here you can use Mastra as normal. All agents, workflows, and tool calls will be automatically traced.
Example Agent Walkthrough
Here is a full project example to get you started:
Launch Phoenix using one of the methods above
The rest of this tutorial will assume you are running Phoenix locally on the default localhost:6006 port.
Create a new Mastra project
npm create mastra@latest
# answer the prompts, include agent, tools, and the example when asked
cd chosen-project-name
npm install --save @arizeai/openinference-mastra
Connect to Phoenix
Add the OpenInference telemetry code to your index.js file. The complete file should now look like this:
Model Details: Model name, provider, token usage, response metadata
Performance Metrics: Execution time, token counts, costs
User Context: Session IDs, user information (if provided)
You can view all of this information in the Phoenix UI to debug issues, optimize performance, and understand your application's behavior.
Model Context Protocol
Anthropic's Model Context Protocol is a standard for connecting AI assistants to the systems where data lives, including content repositories, business tools, and development environments.
Trace your MCP Clients and Servers in a unified view
Phoenix MCP Server
Phoenix MCP Server is an implementation of the Model Context Protocol for the Arize Phoenix platform. It provides a unified interface to Phoenix's capabilites.
Phoenix MCP Server supports:
Prompts Management: Create, list, update, and iterate on prompts
Datasets: Explore datasets, and synthesize new examples
Experiments: Pull experiment results and visualize them with the help of an LLM
Connecting the Phoenix MCP Server
Using the Phoenix MCP Server
The MCP server can be used to interact with Prompts, Experiments, and Datasets. It can be used to retrieve information about each item, and can create and update Prompts.
Some good questions to try:
What prompts do I have in Phoenix?
Create a new prompt in Phoenix that classifies user intent
Update my classification prompt in Phoenix with these new options
Summarize the Phoenix experiments run on my agent inputs dataset
Visualize the results of my jailbreak dataset experiments in Phoenix
Hoping to see additional functionality?
@arizeai/phoenix-mcp is open-source! Issues and PRs welcome.
MCP Tracing
Phoenix provides tracing for MCP clients and servers through OpenInference. This includes the unique capability to trace client to server interactions under a single trace in the correct hierarchy.
The openinference-instrumentation-mcp instrumentor is unique compared to other OpenInference instrumentors. It does not generate any of its own telemetry. Instead, it enables context propagation between MCP clients and servers to unify traces. You still need generate OpenTelemetry traces in both the client and server to see a unified trace.
Install
pip install openinference-instrumentation-mcp
Because the MCP instrumentor does not generate its own telemetry, you must use it alongside other instrumentation code to see traces.
The example code below uses OpenAI agents, which you can instrument using:
import asyncio
from agents import Agent, Runner
from agents.mcp import MCPServer, MCPServerStdio
from dotenv import load_dotenv
from phoenix.otel import register
load_dotenv()
# Connect to your Phoenix instance
tracer_provider = register(auto_instrument=True)
async def run(mcp_server: MCPServer):
agent = Agent(
name="Assistant",
instructions="Use the tools to answer the users question.",
mcp_servers=[mcp_server],
)
while True:
message = input("\n\nEnter your question (or 'exit' to quit): ")
if message.lower() == "exit" or message.lower() == "q":
break
print(f"\n\nRunning: {message}")
result = await Runner.run(starting_agent=agent, input=message)
print(result.final_output)
async def main():
async with MCPServerStdio(
name="Financial Analysis Server",
params={
"command": "fastmcp",
"args": ["run", "./server.py"],
},
client_session_timeout_seconds=30,
) as server:
await run(server)
if __name__ == "__main__":
asyncio.run(main())
Add Tracing to your MCP Server
import json
import os
from datetime import datetime, timedelta
import openai
from dotenv import load_dotenv
from mcp.server.fastmcp import FastMCP
from pydantic import BaseModel
from phoenix.otel import register
load_dotenv()
# You must also connect your MCP server to Phoenix
tracer_provider = register(auto_instrument=True)
# Get a tracer to add additional instrumentattion
tracer = tracer_provider.get_tracer("financial-analysis-server")
# Configure OpenAI client
client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
MODEL = "gpt-4-turbo"
# Create MCP server
mcp = FastMCP("Financial Analysis Server")
class StockAnalysisRequest(BaseModel):
ticker: str
time_period: str = "short-term" # short-term, medium-term, long-term
@mcp.tool()
@tracer.tool(name="MCP.analyze_stock") # this OpenInference call adds tracing to this method
def analyze_stock(request: StockAnalysisRequest) -> dict:
"""Analyzes a stock based on its ticker symbol and provides investment recommendations."""
# Make LLM API call to analyze the stock
prompt = f"""
Provide a detailed financial analysis for the stock ticker: {request.ticker}
Time horizon: {request.time_period}
Please include:
1. Company overview
2. Recent financial performance
3. Key metrics (P/E ratio, market cap, etc.)
4. Risk assessment
5. Investment recommendation
Format your response as a JSON object with the following structure:
{{
"ticker": "{request.ticker}",
"company_name": "Full company name",
"overview": "Brief company description",
"financial_performance": "Analysis of recent performance",
"key_metrics": {{
"market_cap": "Value in billions",
"pe_ratio": "Current P/E ratio",
"dividend_yield": "Current yield percentage",
"52_week_high": "Value",
"52_week_low": "Value"
}},
"risk_assessment": "Analysis of risks",
"recommendation": "Buy/Hold/Sell recommendation with explanation",
"time_horizon": "{request.time_period}"
}}
"""
response = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
)
analysis = json.loads(response.choices[0].message.content)
return analysis
# ... define any additional MCP tools you wish
if __name__ == "__main__":
mcp.run()
Observe
Now that you have tracing setup, all invocations of your client and server will be streamed to Phoenix for observability and evaluation, and connected in the platform.
Portkey is an AI Gateway and observability platform that provides routing, guardrails, caching, and monitoring for 200+ LLMs with enterprise-grade security and reliability features.
How to trace Portkey AI Gateway requests with Phoenix for comprehensive LLM observability
Phoenix provides seamless integration with Portkey, the AI Gateway and observability platform that routes to 200+ LLMs with enterprise-grade features including guardrails, caching, and load balancing.
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see Terminal Setup.
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Use the register function to connect your application to Phoenix:
from phoenix.otel import register
# configure the Phoenix tracer
tracer_provider = register(
project_name="my-portkey-app", # Default is 'default'
auto_instrument=True # Auto-instrument your app based on installed OI dependencies
)
Run Portkey
By instrumenting Portkey, spans will be created whenever requests are made through the AI Gateway and will be sent to the Phoenix server for collection.
Basic Usage with OpenAI
import os
from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
# Set up your API keys
os.environ["OPENAI_API_KEY"] = "your-openai-api-key"
os.environ["PORTKEY_API_KEY"] = "your-portkey-api-key" # Optional for self-hosted
client = OpenAI(
api_key=os.environ.get("OPENAI_API_KEY"),
base_url=PORTKEY_GATEWAY_URL,
default_headers=createHeaders(
provider="openai",
api_key=os.environ.get("PORTKEY_API_KEY")
)
)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "What is artificial intelligence?"}]
)
print(response.choices[0].message.content)
Using Portkey SDK Directly
from portkey_ai import Portkey
# Initialize Portkey client
portkey = Portkey(
api_key="your-portkey-api-key", # Optional for self-hosted
virtual_key="your-openai-virtual-key" # Or use provider-specific virtual keys
)
response = portkey.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Explain machine learning"}]
)
print(response.choices[0].message.content)
Observe
Now that you have tracing setup, all requests through Portkey's AI Gateway will be streamed to your running Phoenix instance for observability and evaluation. You'll be able to see:
Request/Response Traces: Complete visibility into LLM interactions
Routing Decisions: Which provider was selected and why
Fallback Events: When and why fallbacks were triggered
Cache Performance: Hit/miss rates and response times
Cost Tracking: Token usage and costs across providers
Latency Metrics: Response times for each provider and route
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your dashboard.
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see Terminal Setup.
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
For more info on using Phoenix with Docker, see Docker.
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See self-hosting or use one of the other deployment options to retain traces.
Install
pip install promptflow
Setup
Set up the OpenTelemetry endpoint to point to Phoenix and use Prompt flow's setup_exporter_from_environ to start tracing any further flows and LLM calls.
import os
from opentelemetry.sdk.environment_variables import OTEL_EXPORTER_OTLP_ENDPOINT
from promptflow.tracing._start_trace import setup_exporter_from_environ
endpoint = f"{os.environ["PHOENIX_COLLECTOR_ENDPOINT]}/v1/traces" # replace with your Phoenix endpoint if self-hosting
os.environ[OTEL_EXPORTER_OTLP_ENDPOINT] = endpoint
setup_exporter_from_environ()
Run PromptFlow
Proceed with creating Prompt flow flows as usual. See this example notebook for inspiration.
Observe
You should see the spans render in Phoenix as shown in the below screenshots.
PydanticAI is a Python agent framework designed to make it less painful to build production-grade applications with Generative AI, built by the team behind Pydantic with type-safe structured outputs
Set up tracing using OpenTelemetry and the PydanticAI instrumentation:
import os
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from openinference.instrumentation.pydantic_ai import OpenInferenceSpanProcessor
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
# Set up the tracer provider
tracer_provider = TracerProvider()
trace.set_tracer_provider(tracer_provider)
# Add the OpenInference span processor
endpoint = f"{os.environ['PHOENIX_COLLECTOR_ENDPOINT']}/v1/traces"
exporter = OTLPSpanExporter(endpoint=endpoint)
tracer_provider.add_span_processor(OpenInferenceSpanProcessor())
tracer_provider.add_span_processor(SimpleSpanProcessor(exporter))
Basic Usage
Here's a simple example using PydanticAI with automatic tracing:
from pydantic import BaseModel
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
import nest_asyncio
nest_asyncio.apply()
# Set your OpenAI API key
os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"
# Define your Pydantic model
class LocationModel(BaseModel):
city: str
country: str
# Create and configure the agent
model = OpenAIModel("gpt-4", provider='openai')
agent = Agent(model, output_type=LocationModel, instrument=True)
# Run the agent
result = agent.run_sync("The windy city in the US of A.")
print(result)
Advanced Usage
Agent with System Prompts and Tools
from pydantic import BaseModel, Field
from pydantic_ai import Agent, RunContext
from pydantic_ai.models.openai import OpenAIModel
from typing import List
import httpx
class WeatherInfo(BaseModel):
location: str
temperature: float = Field(description="Temperature in Celsius")
condition: str
humidity: int = Field(description="Humidity percentage")
# Create an agent with system prompts and tools
weather_agent = Agent(
model=OpenAIModel("gpt-4"),
output_type=WeatherInfo,
system_prompt="You are a helpful weather assistant. Always provide accurate weather information.",
instrument=True
)
@weather_agent.tool
async def get_weather_data(ctx: RunContext[None], location: str) -> str:
"""Get current weather data for a location."""
# Mock weather API call - replace with actual weather service
async with httpx.AsyncClient() as client:
# This is a placeholder - use a real weather API
mock_data = {
"temperature": 22.5,
"condition": "partly cloudy",
"humidity": 65
}
return f"Weather in {location}: {mock_data}"
# Run the agent with tool usage
result = weather_agent.run_sync("What's the weather like in Paris?")
print(result)
Observe
Now that you have tracing setup, all PydanticAI agent operations will be streamed to your running Phoenix instance for observability and evaluation. You'll be able to see:
Agent interactions: Complete conversations between your application and the AI model
Structured outputs: Pydantic model validation and parsing results
Tool usage: When agents call external tools and their responses
Performance metrics: Response times, token usage, and success rates
Error handling: Validation errors, API failures, and retry attempts
Multi-agent workflows: Complex interactions between multiple agents
The traces will provide detailed insights into your AI agent behaviors, making it easier to debug issues, optimize performance, and ensure reliability in production.
Resources
Loading...
Vercel
Vercel is a cloud platform that simplifies building, deploying, and scaling modern web applications with features like serverless functions, edge caching, and seamless Git integration
This package provides a set of utilities to ingest Vercel AI SDK(>= 3.3) spans into platforms like Arize and Phoenix.
Note: This package requires you to be using the Vercel AI SDK version 3.3 or higher.
Installation
npm install --save @arizeai/openinference-vercel
You will also need to install OpenTelemetry and Vercel packages to your project.
npm i @opentelemetry/api @vercel/otel @opentelemetry/exporter-trace-otlp-proto @arizeai/openinference-semantic-conventions
Usage
@arizeai/openinference-vercel provides a set of utilities to help you ingest Vercel AI SDK spans into platforms and works in conjunction with Vercel's OpenTelemetry support. To get started, you will need to add OpenTelemetry support to your Vercel project according to their guide.
To process your Vercel AI SDK Spans add a OpenInferenceSimpleSpanProcessor or OpenInferenceBatchSpanProcessor to your OpenTelemetry configuration.
Note: The OpenInferenceSpanProcessor does not handle the exporting of spans so you will need to pass it an exporter as a parameter.
import { registerOTel } from "@vercel/otel";
import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api";
import {
isOpenInferenceSpan,
OpenInferenceSimpleSpanProcessor,
} from "@arizeai/openinference-vercel";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { SEMRESATTRS_PROJECT_NAME } from "@arizeai/openinference-semantic-conventions";
// For troubleshooting, set the log level to DiagLogLevel.DEBUG
// This is not required and should not be added in a production setting
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);
export function register() {
registerOTel({
serviceName: "phoenix-next-app",
attributes: {
// This is not required but it will allow you to send traces to a specific
// project in phoenix
[SEMRESATTRS_PROJECT_NAME]: "your-next-app",
},
spanProcessors: [
new OpenInferenceSimpleSpanProcessor({
exporter: new OTLPTraceExporter({
headers: {
// API key if you're sending it to Phoenix
api_key: process.env["PHOENIX_API_KEY"],
},
url:
process.env["PHOENIX_COLLECTOR_ENDPOINT"] ||
"https://app.phoenix.arize.com/v1/traces",
}),
spanFilter: (span) => {
// Only export spans that are OpenInference to remove non-generative spans
// This should be removed if you want to export all spans
return isOpenInferenceSpan(span);
},
}),
],
});
}
Now enable telemetry in your AI SDK calls by setting the experimental_telemetry parameter to true.
const result = await generateText({
model: openai("gpt-4-turbo"),
prompt: "Write a short story about a cat.",
experimental_telemetry: { isEnabled: true },
});
Ensuring the reliability and accuracy of LLM-generated responses is a critical challenge for production AI systems. Poor-quality training data, ambiguous labels, and untrustworthy outputs can degrade model performance and lead to unreliable results.
Cleanlab TLM is a tool that estimates the trustworthiness of an LLM response. It provides a confidence score that helps detect hallucinations, ambiguous responses, and potential misinterpretations. This enables teams to flag unreliable outputs and improve the robustness of their AI systems.
This guide demonstrates how to integrate Cleanlab’s Trustworthy Language Model (TLM) with Phoenix to systematically identify and improve low-quality LLM responses. By leveraging TLM for automated data quality assessment and Phoenix for response analysis, you can build more robust and trustworthy AI applications.
Specifically, this tutorial will walk through:
Evaluating LLM-generated responses for trustworthiness.
Using Cleanlab TLM to score and flag untrustworthy responses.
Leveraging Phoenix for tracing and visualizing response evaluations.
Key Implementation Steps for generating evals w/ TLM
Install Dependencies, Set up API Keys, Obtain LLM Reponses + Trace in Phoenix
Download Trace Dataset
import phoenix as px
spans_df = px.Client().get_spans_dataframe(project_name=[your_project_name])
spans_df.head()
Prep data from trace dataset
# Create a new DataFrame with input and output columns
eval_df = spans_df[["context.span_id", "attributes.input.value", "attributes.output.value"]].copy()
eval_df.set_index("context.span_id", inplace=True)
# Combine system and user prompts from the traces
def get_prompt(input_value):
if isinstance(input_value, str):
input_value = json.loads(input_value)
system_prompt = input_value["messages"][0]["content"]
user_prompt = input_value["messages"][1]["content"]
return system_prompt + "\n" + user_prompt
# Get the responses from the traces
def get_response(output_value):
if isinstance(output_value, str):
output_value = json.loads(output_value)
return output_value["choices"][0]["message"]["content"]
# Create a list of prompts and associated responses
prompts = [get_prompt(input_value) for input_value in eval_df["attributes.input.value"]]
responses = [get_response(output_value) for output_value in eval_df["attributes.output.value"]]
eval_df["prompt"] = prompts
eval_df["response"] = responses
Setup TLM & Evaluate each pair
from cleanlab_tlm import TLM
tlm = TLM(options={"log": ["explanation"]})
# Evaluate each of the prompt, response pairs using TLM
evaluations = tlm.get_trustworthiness_score(prompts, responses)
# Extract the trustworthiness scores and explanations from the evaluations
trust_scores = [entry["trustworthiness_score"] for entry in evaluations]
explanations = [entry["log"]["explanation"] for entry in evaluations]
# Add the trust scores and explanations to the DataFrame
eval_df["score"] = trust_scores
eval_df["explanation"] = explanations
Ragas is a library that provides robust evaluation metrics for LLM applications, making it easy to assess quality. When integrated with Phoenix, it enriches your experiments with metrics like goal accuracy and tool call accuracy—helping you evaluate performance more effectively and track improvements over time.
This guide will walk you through the process of creating and evaluating agents using Ragas and Arize Phoenix. We'll cover the following steps:
Build a customer support agent with the OpenAI Agents SDK
Trace agent activity to monitor interactions
Generate a benchmark dataset for performance analysis
Evaluate agent performance using Ragas
We will walk through the key steps in the documentation below. Check out the full tutorial here:
Creating the Agent
Here we've setup a basic agent that can solve math problems. We have a function tool that can solve math equations, and an agent that can use this tool. We'll use the Runner class to run the agent and get the final output.
from agents import Runner, function_tool
@function_tool
def solve_equation(equation: str) -> str:
"""Use python to evaluate the math equation, instead of thinking about it yourself.
Args:"
equation: string which to pass into eval() in python
"""
return str(eval(equation))
from agents import Agent
agent = Agent(
name="Math Solver",
instructions="You solve math problems by evaluating them with python and returning the result",
tools=[solve_equation],
)
Evaluating the Agent
Agents can go awry for a variety of reasons. We can use Ragas to evaluate whether the agent responded correctly. Two Ragas measurements help with this:
Tool Call Accuracy - Did our agent choose the right tool with the right arguments?
Agent Goal Accuracy - Did our agent accomplish the stated goal and get to the right outcome?
We'll import both metrics we're measuring from Ragas, and use the multi_turn_ascore(sample) to get the results. The AgentGoalAccuracyWithReference metric compares the final output to the reference to see if the goal was accomplished. The ToolCallAccuracy metric compares the tool call to the reference tool call to see if the tool call was made correctly.
In the notebook, we also define the helper function conversation_to_ragas_sample which converts the agent messages into a format that Ragas can use.
The following code snippets define our task function and evaluators.
import asyncio
from agents import Runner
async def solve_math_problem(input):
if isinstance(input, dict):
input = next(iter(input.values()))
result = await Runner.run(agent, input)
return {"final_output": result.final_output, "messages": result.to_input_list()}
Phoenix can be used to trace and evaluate applications that use Weaviate as a vector database.
Examples
Ingesting Data for Semantic Search
This tutorial will show you how to embed a large volume of data, upload it to a vector database, run top K similarity searches against it, and monitor it in production using VectorFlow, Arize Phoenix, Weaviate and LlamaIndex.
Instrumenting and Evaluating a Weaviate RAG Pipeline
This tutorial serves as a great starting point to see how to manually instrument a RAG chatbot built on Weaviate, and visualize and evaluate the results in Phoenix.
Zilliz / Milvus
Milvus is an open-source vector database built for GenAI applications.
Your Phoenix API key can be found on the Keys section of your .
For details on customizing a local terminal deployment, see .
Pull latest Phoenix image from:
For more info on using Phoenix with Docker, see .
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See or use one of the other deployment options to retain traces.
Traces and spans from an instrumented DSPy custom module.
From the Cursor Settings page, navigate to the MCP section, and click "Add new global MCP server"
is a Python agent framework designed to make it less painful to build production-grade applications with Generative AI. Built by the team behind Pydantic, it provides a clean, type-safe way to build AI agents with structured outputs.
Sign up for Phoenix:
Sign up for an Arize Phoenix account at
Install packages:
pipinstallarize-phoenix-otel
Set your Phoenix endpoint and API Key:
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your .
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see .
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See or use one of the other deployment options to retain traces.
How to use Pydantic Evals with Phoenix to evaluate AI applications using structured evaluation frameworks
Launch Phoenix
Install
Setup
Enable Phoenix tracing to capture traces from your application:
Basic Usage
1. Generate Traces to Evaluate
First, create some example traces by running your AI application. Here's a simple example:
2. Export Traces from Phoenix
Export the traces you want to evaluate:
3. Define Evaluation Dataset
Create a dataset of test cases using Pydantic Evals:
4. Create Custom Evaluators
Define evaluators to assess your model's performance:
5. Setup Task and Dataset
Create a task that retrieves outputs from your traced data:
6. Add LLM Judge Evaluator
For more sophisticated evaluation, add an LLM judge:
7. Run Evaluation
Execute the evaluation:
Advanced Usage
Upload Results to Phoenix
Upload your evaluation results back to Phoenix for visualization:
Custom Evaluation Workflows
You can create more complex evaluation workflows by combining multiple evaluators:
Observe
Once you have evaluation results uploaded to Phoenix, you can:
View evaluation metrics: See overall performance across different evaluation criteria
Analyze individual cases: Drill down into specific examples that passed or failed
Compare evaluators: Understand how different evaluation methods perform
Track improvements: Monitor evaluation scores over time as you improve your application
Debug failures: Identify patterns in failed evaluations to guide improvements
The Phoenix UI will display your evaluation results with detailed breakdowns, making it easy to understand your AI application's performance and identify areas for improvement.
is an evaluation library that provides preset direct evaluations and LLM Judge evaluations. It can be used to run evaluations over dataframes of cases defined with Pydantic models. This guide shows you how to use Pydantic Evals alongside Arize Phoenix to run evaluations on traces captured from your running application.
Sign up for Phoenix:
Sign up for an Arize Phoenix account at
Install packages:
Set your Phoenix endpoint and API Key:
Your Phoenix API key can be found on the Keys section of your .
Launch your local Phoenix instance:
For details on customizing a local terminal deployment, see .
Install packages:
Set your Phoenix endpoint:
See Terminal for more details
Run your containerized instance:
This will expose the Phoenix on localhost:6006
Install packages:
Set your Phoenix endpoint:
Install packages:
Launch Phoenix:
Pull latest Phoenix image from:
For more info on using Phoenix with Docker, see .
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See or use one of the other deployment options to retain traces.
from openai import OpenAI
import os
client = OpenAI()
inputs = [
"What is the capital of France?",
"Who wrote Romeo and Juliet?",
"What is the largest planet in our solar system?",
]
def generate_trace(input):
client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "system",
"content": "You are a helpful assistant. Only respond with the answer to the question as a single word or proper noun.",
},
{"role": "user", "content": input},
],
)
for input in inputs:
generate_trace(input)
from pydantic_evals import Case, Dataset
cases = [
Case(
name="capital of France",
inputs="What is the capital of France?",
expected_output="Paris"
),
Case(
name="author of Romeo and Juliet",
inputs="Who wrote Romeo and Juliet?",
expected_output="William Shakespeare",
),
Case(
name="largest planet",
inputs="What is the largest planet in our solar system?",
expected_output="Jupiter",
),
]
from pydantic_evals.evaluators import Evaluator, EvaluatorContext
class MatchesExpectedOutput(Evaluator[str, str]):
def evaluate(self, ctx: EvaluatorContext[str, str]) -> float:
is_correct = ctx.expected_output == ctx.output
return is_correct
class FuzzyMatchesOutput(Evaluator[str, str]):
def evaluate(self, ctx: EvaluatorContext[str, str]) -> float:
from difflib import SequenceMatcher
def similarity_ratio(a, b):
return SequenceMatcher(None, a, b).ratio()
# Consider it correct if similarity is above 0.8 (80%)
is_correct = similarity_ratio(ctx.expected_output, ctx.output) > 0.8
return is_correct
from pydantic_evals.evaluators import LLMJudge
dataset.add_evaluator(
LLMJudge(
rubric="Output and Expected Output should represent the same answer, even if the text doesn't match exactly",
include_input=True,
model="openai:gpt-4o-mini",
),
)
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your .
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see .
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
Sign up for Phoenix:
Sign up for an Arize Phoenix account at
Install packages:
pipinstallarize-phoenix-otel
Set your Phoenix endpoint and API Key:
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your .
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see .
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
Sign up for Phoenix:
Sign up for an Arize Phoenix account at
Install packages:
Set your Phoenix endpoint and API Key:
Your Phoenix API key can be found on the Keys section of your .
Launch your local Phoenix instance:
For details on customizing a local terminal deployment, see .
Install packages:
Set your Phoenix endpoint:
See for more details.
Run your containerized instance:
This will expose the Phoenix on localhost:6006
Install packages:
Set your Phoenix endpoint:
Install packages:
Launch Phoenix:
Sign up for Phoenix:
Sign up for an Arize Phoenix account at
Install packages:
Set your Phoenix endpoint and API Key:
Your Phoenix API key can be found on the Keys section of your .
Launch your local Phoenix instance:
For details on customizing a local terminal deployment, see .
Install packages:
Set your Phoenix endpoint:
See for more details.
Run your containerized instance:
This will expose the Phoenix on localhost:6006
Install packages:
Set your Phoenix endpoint:
Install packages:
Launch Phoenix:
Sign up for Phoenix:
Sign up for an Arize Phoenix account at
Install packages:
Set your Phoenix endpoint and API Key:
Your Phoenix API key can be found on the Keys section of your .
Launch your local Phoenix instance:
For details on customizing a local terminal deployment, see .
Install packages:
Set your Phoenix endpoint:
See for more details.
Run your containerized instance:
This will expose the Phoenix on localhost:6006
Install packages:
Set your Phoenix endpoint:
Install packages:
Launch Phoenix:
Pull latest Phoenix image from:
For more info on using Phoenix with Docker, see .
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See or use one of the other deployment options to retain traces.
Pull latest Phoenix image from:
For more info on using Phoenix with Docker, see .
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See or use one of the other deployment options to retain traces.
Pull latest Phoenix image from:
For more info on using Phoenix with Docker, see .
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See or use one of the other deployment options to retain traces.
Pull latest Phoenix image from:
For more info on using Phoenix with Docker, see .
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See or use one of the other deployment options to retain traces.
Pull latest Phoenix image from:
For more info on using Phoenix with Docker, see .
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See or use one of the other deployment options to retain traces.
docker run -p 6006:6006 arizephoenix/phoenix:latest
pip install arize-phoenix-otel
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
pip install arize-phoenix
import phoenix as px
px.launch_app()
docker pull arizephoenix/phoenix:latest
docker run -p 6006:6006 arizephoenix/phoenix:latest
pip install arize-phoenix-otel
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
pip install arize-phoenix
import phoenix as px
px.launch_app()
docker pull arizephoenix/phoenix:latest
docker run -p 6006:6006 arizephoenix/phoenix:latest
pip install arize-phoenix-otel
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
pip install arize-phoenix
import phoenix as px
px.launch_app()
pipinstallarize-phoenix-otel
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your .
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see .
Your Phoenix API key can be found on the Keys section of your .
Launch your local Phoenix instance:
This will expose the Phoenix on localhost:6006
For details on customizing a local terminal deployment, see .
Set your Phoenix endpoint and API Key:
Pull latest Phoenix image from:
Run your containerized instance:
This will expose the Phoenix on localhost:6006
Set your Phoenix endpoint and API Key:
For more info on using Phoenix with Docker, see .
Pull latest Phoenix image from:
For more info on using Phoenix with Docker, see .
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See or use one of the other deployment options to retain traces.
export PHOENIX_COLLECTOR_ENDPOINT="http://localhost:6006/v1/traces"export PHOENIX_API_KEY="YOUR PHOENIX API KEY"# only necessary if you've enabled auth
dockerpullarizephoenix/phoenix:latest
dockerrun-p6006:6006arizephoenix/phoenix:latest
export PHOENIX_COLLECTOR_ENDPOINT="http://localhost:6006/v1/traces"export PHOENIX_API_KEY="YOUR PHOENIX API KEY"# only necessary if you've enabled auth
import os# Add Phoenix API Key for tracingPHOENIX_API_KEY ="ADD YOUR API KEY"os.environ["PHOENIX_CLIENT_HEADERS"]=f"api_key={PHOENIX_API_KEY}"os.environ["PHOENIX_COLLECTOR_ENDPOINT"]="https://app.phoenix.arize.com"
Your Phoenix API key can be found on the Keys section of your .
Launch your local Phoenix instance:
pipinstallarize-phoenixphoenixserve
For details on customizing a local terminal deployment, see .
docker run -p 6006:6006 arizephoenix/phoenix:latest
This will expose the Phoenix on localhost:6006
Install packages:
pip install arize-phoenix-otel
Set your Phoenix endpoint:
import os
os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "http://localhost:6006"
Install packages:
pip install arize-phoenix
Launch Phoenix:
import phoenix as px
px.launch_app()
Pull latest Phoenix image from:
For more info on using Phoenix with Docker, see .
By default, notebook instances do not have persistent storage, so your traces will disappear after the notebook is closed. See or use one of the other deployment options to retain traces.
