The following are simple functions on top of the LLM evals building blocks that are pre-tested with benchmark data.

Supported Models.

The models are instantiated and usable in the LLM Eval function. The models are also directly callable with strings.

model = OpenAIModel(model_name="gpt-4",temperature=0.6)
model("What is the largest costal city in France?")

We currently support a growing set of models for LLM Evals, please check out the Eval Models section for usage.

When To Use Hallucination Eval Template

This LLM Eval detects if the output of a model is a hallucination based on contextual data.

This Eval is specifically designed to detect hallucinations in generated answers from private or retrieved data. The Eval detects if an AI answer to a question is a hallucination based on the reference data used to generate the answer.

Hallucination Eval Template

How To Run the Hallucination Eval

The above Eval shows how to the the hallucination template for Eval detection.

Benchmark Results

This benchmark was obtained using notebook below. It was run using the as a ground truth dataset. Each example in the dataset was evaluating using the HALLUCINATION_PROMPT_TEMPLATE above, then the resulting labels were compared against the is_hallucination label in the HaluEval dataset to generate the confusion matrices below.

GPT-4 Results

Phoenix Evaluators

• Hallucinations

• Q&A on Retrieved Data

• Retrieval (RAG) Relevance

• Summarization

• Code Generation

• Toxicity

• AI vs Human

• Reference (Citation) Eval

• User Frustration

• SQL Generation Eval

• Agent Function Calling Eval

• Audio Emotion Detection

Bring Your Own Evaluator

• Categorical evaluator (llm_classify)

• Numeric evaluator (llm_generate)

Online Evals

Run evaluations via a job to visualize in the UI as traces stream in.

Evaluating Phoenix Traces

Evaluate traces captured in Phoenix and export results to the Phoenix UI.

Multimodal Evals

Evaluate tasks with multiple inputs/outputs (ex: text, audio, image) using versatile evaluation tasks.

The Emotion Detection Eval Template is designed to classify emotions from audio files. This evaluation leverages predefined characteristics, such as tone, pitch, and intensity, to detect the most dominant emotion expressed in an audio input. This guide will walk you through how to use the template within the Phoenix framework to evaluate emotion classification models effectively.

Template Details

The following is the structure of the EMOTION_PROMPT_TEMPLATE:

You are an AI system designed to classify emotions in audio files.

### TASK:
Analyze the provided audio file and classify the primary emotion based on these characteristics:
- Tone: General tone of the speaker (e.g., cheerful, tense, calm).
- Pitch: Level and variability of the pitch (e.g., high, low, monotone).
- Pace: Speed of speech (e.g., fast, slow, steady).
- Volume: Loudness of the speech (e.g., loud, soft, moderate).
- Intensity: Emotional strength or expression (e.g., subdued, sharp, exaggerated).

The classified emotion must be one of the following:
['anger', 'happiness', 'excitement', 'sadness', 'neutral', 'frustration', 'fear', 'surprise', 'disgust', 'other']

IMPORTANT: Choose the most dominant emotion expressed in the audio. Neutral should only be used when no other emotion is clearly present; do your best to avoid this label.

************

Here is the audio to classify:

{audio}

RESPONSE FORMAT:

Provide a single word from the list above representing the detected emotion.

************

EXAMPLE RESPONSE: excitement

************

Analyze the audio and respond in this format.

Template Module

The prompt and evaluation logic are part of the phoenix.evals.default_audio_templates module and are defined as:

• EMOTION_AUDIO_RAILS: Output options for the evaluation template.

• EMOTION_PROMPT_TEMPLATE: Prompt used for evaluating audio emotions.

This template evaluates a plan generated by an agent. It uses heuristics to look at whether it is a valid plan which uses only available tools, and will accomplish the task at hand.

Prompt Template

You are an evaluation assistant. Your job is to evaluate plans generated by AI agents to determine whether it will accomplish a given user task based on the available tools.

Here is the data:
    [BEGIN DATA]
    ************
    [User task]: {task}
    ************
    [Tools]: {tool_definitions}
    ************
    [Plan]: {plan}
    [END DATA]

Here is the criteria for evaluation
1. Does the plan include only valid and applicable tools for the task?  
2. Are the tools used in the plan sufficient to accomplish the task?  
3. Will the plan, as outlined, successfully achieve the desired outcome?  
4. Is this the shortest and most efficient plan to accomplish the task?

Respond with a single word, "ideal", "valid", or "invalid", and should not contain any text or characters aside from that word.

"ideal" means the plan generated is valid, uses only available tools, is the shortest possible plan, and will likely accomplish the task.

"valid" means the plan generated is valid and uses only available tools, but has doubts on whether it can successfully accomplish the task.

"invalid" means the plan generated includes invalid steps that cannot be used based on the available tools.

Use this prompt template to evaluate an agent's final response. This is an optional step, which you can use as a gate to retry a set of actions if the response or state of the world is insufficient for the given task.

Read more:

• How to evaluate the evaluators and build self-improving evals

• Prompt optimization

Prompt Template

This prompt template is heavily inspired by the paper: "Self Reflection in LLM Agents".

You are an expert in {topic}. I will give you a user query. Your task is to reflect on your provided solution and whether it has solved the problem.
First, explain whether you believe the solution is correct or incorrect.
Second, list the keywords that describe the type of your errors from most general to most specific.
Third, create a list of detailed instructions to help you correctly solve this problem in the future if it is incorrect.

Be concise in your response; however, capture all of the essential information.

Here is the data:
    [BEGIN DATA]
    ************
    [User Query]: {user_query}
    ************
    [Tools]: {tool_definitions}
    ************
    [State]: {current_state}
    ************
    [Provided Solution]: {solution}
    [END DATA]

When your agents take multiple steps to get to an answer or resolution, it's important to evaluate the pathway it took to get there. You want most of your runs to be consistent and not take unnecessarily frivolous or wrong actions.

One way of doing this is to calculate convergence:

1. Run your agent on a set of similar queries

2. Record the number of steps taken for each

3. Calculate the convergence score: avg(minimum steps taken / steps taken for this run)

This will give a convergence score of 0-1, with 1 being a perfect score.

# Assume you have an output which has a list of messages, which is the path taken
all_outputs = [
]

optimal_path_length = min(all_outputs, key = lambda output: len(output))
ratios_sum = 0

for output in all_outputs:
    run_length = len(output)
    ratio = optimal_path_length / run_length
    ratios_sum += ratio

# Calculate the average ratio
if len(all_outputs) > 0:
    convergence = ratios_sum / len(all_outputs)
else:
    convergence = 0

print(f"The optimal path length is {optimal_path_length}")
print(f"The convergence is {convergence}")

SQL Generation is a common approach to using an LLM. In many cases the goal is to take a human description of the query and generate matching SQL to the human description.

Example of a Question: How many artists have names longer than 10 characters?

Example Query Generated:

SELECT COUNT(ArtistId) \nFROM artists \nWHERE LENGTH(Name) > 10

The goal of the SQL generation Evaluation is to determine if the SQL generated is correct based on the question asked.

SQL Eval Template

Running an SQL Generation Eval

Teams that are using conversation bots and assistants desire to know whether a user interacting with the bot is frustrated. The user frustration evaluation can be used on a single back and forth or an entire span to detect whether a user has become frustrated by the conversation.

User Frustration Eval Template

The following is an example of code snippet showing how to use the eval above template:

A workflow we see for high quality RAG deployments is generating a golden dataset of questions and a high quality set of answers. These can be in the range of 100-200 but provide a strong check for the AI generated answers. This Eval checks that the human ground truth matches the AI generated answer. Its designed to catch missing data in "half" answers and differences of substance.

Example Human vs AI on Arize Docs:

Question:

What Evals are supported for LLMs on generative models?

Human:

Arize supports a suite of Evals available from the Phoenix Evals library, they include both pre-tested Evals and the ability to configure cusotm Evals. Some of the pre-tested LLM Evals are listed below:

Retrieval Relevance, Question and Answer, Toxicity, Human Groundtruth vs AI, Citation Reference Link Relevancy, Code Readability, Code Execution, Hallucination Detection and Summarizaiton

AI:

Arize supports LLM Evals.

Eval:

Incorrect

Explanation of Eval:

The AI answer is very brief and lacks the specific details that are present in the human ground truth answer. While the AI answer is not incorrect in stating that Arize supports LLM Evals, it fails to mention the specific types of Evals that are supported, such as Retrieval Relevance, Question and Answer, Toxicity, Human Groundtruth vs AI, Citation Reference Link Relevancy, Code Readability, Hallucination Detection, and Summarization. Therefore, the AI answer does not fully capture the substance of the human answer.

Overview of template:

print(HUMAN_VS_AI_PROMPT_TEMPLATE)

You are comparing a human ground truth answer from an expert to an answer from an AI model.
Your goal is to determine if the AI answer correctly matches, in substance, the human answer.
    [BEGIN DATA]
    ************
    [Question]: {question}
    ************
    [Human Ground Truth Answer]: {correct_answer}
    ************
    [AI Answer]: {ai_generated_answer}
    ************
    [END DATA]
Compare the AI answer to the human ground truth answer, if the AI correctly answers the question,
then the AI answer is "correct". If the AI answer is longer but contains the main idea of the
Human answer please answer "correct". If the AI answer diverges or does not contain the main
idea of the human answer, please answer "incorrect".

How to run the Human vs AI Eval:

from phoenix.evals import (
    HUMAN_VS_AI_PROMPT_RAILS_MAP,
    HUMAN_VS_AI_PROMPT_TEMPLATE,
    OpenAIModel,
    llm_classify,
)

model = OpenAIModel(
    model_name="gpt-4",
    temperature=0.0,
)

# The rails is used to hold the output to specific values based on the template
# It will remove text such as ",,," or "..."
# Will ensure the binary value expected from the template is returned
rails = list(HUMAN_VS_AI_PROMPT_RAILS_MAP.values())
relevance_classifications = llm_classify(
    dataframe=df,
    template=HUMAN_VS_AI_PROMPT_TEMPLATE,
    model=model,
    rails=rails,
    verbose=False,
    provide_explanation=True
)

Benchmark Results:

The follow benchmarking data was gathered by comparing various model results to ground truth data. The ground truth data used was a handcrafted dataset consisting of questions about the Arize platform. That dataset is availabe here.

GPT-4 Results

When To Use RAG Eval Template

This Eval evaluates whether a retrieved chunk contains an answer to the query. It's extremely useful for evaluating retrieval systems.

RAG Eval Template

You are comparing a reference text to a question and trying to determine if the reference text
contains information relevant to answering the question. Here is the data:
    [BEGIN DATA]
    ************
    [Question]: {query}
    ************
    [Reference text]: {reference}
    [END DATA]

Compare the Question above to the Reference text. You must determine whether the Reference text
contains information that can answer the Question. Please focus on whether the very specific
question can be answered by the information in the Reference text.
Your response must be single word, either "relevant" or "unrelated",
and should not contain any text or characters aside from that word.
"unrelated" means that the reference text does not contain an answer to the Question.
"relevant" means the reference text contains an answer to the Question.

How To Run the RAG Relevance Eval

from phoenix.evals import (
    RAG_RELEVANCY_PROMPT_RAILS_MAP,
    RAG_RELEVANCY_PROMPT_TEMPLATE,
    OpenAIModel,
    download_benchmark_dataset,
    llm_classify,
)

model = OpenAIModel(
    model_name="gpt-4",
    temperature=0.0,
)

#The rails is used to hold the output to specific values based on the template
#It will remove text such as ",,," or "..."
#Will ensure the binary value expected from the template is returned
rails = list(RAG_RELEVANCY_PROMPT_RAILS_MAP.values())
relevance_classifications = llm_classify(
    dataframe=df,
    template=RAG_RELEVANCY_PROMPT_TEMPLATE,
    model=model,
    rails=rails,
    provide_explanation=True, #optional to generate explanations for the value produced by the eval LLM
)

The above runs the RAG relevancy LLM template against the dataframe df.

Benchmark Results

This benchmark was obtained using notebook below. It was run using the WikiQA dataset as a ground truth dataset. Each example in the dataset was evaluating using the RAG_RELEVANCY_PROMPT_TEMPLATE above, then the resulting labels were compared against the ground truth label in the WikiQA dataset to generate the confusion matrices below.

GPT-4 Result

When To Use Q&A Eval Template

This Eval evaluates whether a question was correctly answered by the system based on the retrieved data. In contrast to retrieval Evals that are checks on chunks of data returned, this check is a system level check of a correct Q&A.

• question: This is the question the Q&A system is running against

• sampled_answer: This is the answer from the Q&A system.

• context: This is the context to be used to answer the question, and is what Q&A Eval must use to check the correct answer

Q&A Eval Template

You are given a question, an answer and reference text. You must determine whether the
given answer correctly answers the question based on the reference text. Here is the data:
    [BEGIN DATA]
    ************
    [Question]: {question}
    ************
    [Reference]: {context}
    ************
    [Answer]: {sampled_answer}
    [END DATA]
Your response must be a single word, either "correct" or "incorrect",
and should not contain any text or characters aside from that word.
"correct" means that the question is correctly and fully answered by the answer.
"incorrect" means that the question is not correctly or only partially answered by the
answer.

How To Run the Q&A Eval

import phoenix.evals.templates.default_templates as templates
from phoenix.evals import (
    OpenAIModel,
    download_benchmark_dataset,
    llm_classify,
)

model = OpenAIModel(
    model_name="gpt-4",
    temperature=0.0,
)

#The rails fore the output to specific values of the template
#It will remove text such as ",,," or "...", anything not the
#binary value expected from the template
rails = list(templates.QA_PROMPT_RAILS_MAP.values())
Q_and_A_classifications = llm_classify(
    dataframe=df_sample,
    template=templates.QA_PROMPT_TEMPLATE,
    model=model,
    rails=rails,
    provide_explanation=True, #optional to generate explanations for the value produced by the eval LLM
)

The above Eval uses the QA template for Q&A analysis on retrieved data.

Benchmark Results

The benchmarking dataset used was created based on:

• Squad 2: The 2.0 version of the large-scale dataset Stanford Question Answering Dataset (SQuAD 2.0) allows researchers to design AI models for reading comprehension tasks under challenging constraints. https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/reports/default/15785042.pdf

• Supplemental Data to Squad 2: In order to check the case of detecting incorrect answers, we created wrong answers based on the context data. The wrong answers are intermixed with right answers.

Each example in the dataset was evaluating using the QA_PROMPT_TEMPLATE above, then the resulting labels were compared against the ground truth in the benchmarking dataset.

GPT-4 Results

Reference Links in Retrieval Q&A

In chatbots and Q&A systems, many times reference links are provided in the response, along with an answer, to help point users to documentation or pages that contain more information or the source for the answer.

EXAMPLE: Q&A from Arize-Phoenix Documentation

QUESTION: What other models does Arize Phoenix support beyond OpenAI for running Evals?

ANSWER: Phoenix does support a large set of LLM models through the model object. Phoenix supports OpenAI (GPT-4, GPT-4-32k, GPT-3.5 Turbo, GPT-3.5 Instruct, etc...), Azure OpenAI, Google Palm2 Text Bison, and All AWS Bedrock models (Claude, Mistral, etc...).

REFERENCE LINK: https://arize.com/docs/phoenix/api/evaluation-models

This Eval checks the reference link returned answers the question asked in a conversation

print(REF_LINK_EVAL_PROMPT_TEMPLATE_STR)

You are given a conversation that contains questions by a CUSTOMER and you are trying
to determine if the documentation page shared by the ASSISTANT correctly answers
the CUSTOMERS questions. We will give you the conversation between the customer
and the ASSISTANT and the text of the documentation returned:
    [CONVERSATION AND QUESTION]:
    {conversation}
    ************
    [DOCUMENTATION URL TEXT]:
    {document_text}
    [DOCUMENTATION URL TEXT]:
You should respond "correct" if the documentation text answers the question the
CUSTOMER had in the conversation. If the documentation roughly answers the question
even in a general way the please answer "correct". If there are multiple questions and a single
question is answered, please still answer "correct". If the text does not answer the
question in the conversation, or doesn't contain information that would allow you
to answer the specific question please answer "incorrect".

How to run the Citation Eval

from phoenix.evals import (
    REF_LINK_EVAL_PROMPT_RAILS_MAP,
    REF_LINK_EVAL_PROMPT_TEMPLATE_STR,
    OpenAIModel,
    download_benchmark_dataset,
    llm_classify,
)

model = OpenAIModel(
    model_name="gpt-4",
    temperature=0.0,
)

#The rails is used to hold the output to specific values based on the template
#It will remove text such as ",,," or "..."
#Will ensure the binary value expected from the template is returned
rails = list(REF_LINK_EVAL_PROMPT_RAILS_MAP.values())
relevance_classifications = llm_classify(
    dataframe=df,
    template=REF_LINK_EVAL_PROMPT_TEMPLATE_STR,
    model=model,
    rails=rails,
    provide_explanation=True, #optional to generate explanations for the value produced by the eval LLM
)

Benchmark Results

This benchmark was obtained using notebook below. It was run using a handcrafted ground truth dataset consisting of questions on the Arize platform. That dataset is available here.

Each example in the dataset was evaluating using the REF_LINK_EVAL_PROMPT_TEMPLATE_STR above, then the resulting labels were compared against the ground truth label in the benchmark dataset to generate the confusion matrices below.

GPT-4 Results

The Agent Function Call eval can be used to determine how well a model selects a tool to use, extracts the right parameters from the user query, and generates the tool call code.

Function Calling Eval Template

TOOL_CALLING_PROMPT_TEMPLATE = """
You are an evaluation assistant evaluating questions and tool calls to
determine whether the tool called would answer the question. The tool
calls have been generated by a separate agent, and chosen from the list of
tools provided below. It is your job to decide whether that agent chose
the right tool to call.

    [BEGIN DATA]
    ************
    [Question]: {question}
    ************
    [Tool Called]: {tool_call}
    [END DATA]

Your response must be single word, either "correct" or "incorrect",
and should not contain any text or characters aside from that word.
"incorrect" means that the chosen tool would not answer the question,
the tool includes information that is not presented in the question,
or that the tool signature includes parameter values that don't match
the formats specified in the tool signatures below.

"correct" means the correct tool call was chosen, the correct parameters
were extracted from the question, the tool call generated is runnable and correct,
and that no outside information not present in the question was used
in the generated question.

    [Tool Definitions]: {tool_definitions}
"""

Running an Agent Eval using the Function Calling Template

from phoenix.evals import (
    TOOL_CALLING_PROMPT_RAILS_MAP,
    TOOL_CALLING_PROMPT_TEMPLATE,
    OpenAIModel,
    llm_classify,
)

# the rails object will be used to snap responses to "correct" 
# or "incorrect"
rails = list(TOOL_CALLING_PROMPT_RAILS_MAP.values())
model = OpenAIModel(
    model_name="gpt-4",
    temperature=0.0,
)

# Loop through the specified dataframe and run each row 
# through the specified model and prompt. llm_classify
# will run requests concurrently to improve performance.
tool_call_evaluations = llm_classify(
    dataframe=df,
    template=TOOL_CALLING_PROMPT_TEMPLATE,
    model=model,
    rails=rails,
    provide_explanation=True
)

Parameters:

• df - a dataframe of cases to evaluate. The dataframe must have these columns to match the default template:

  • question - the query made to the model. If you've to evaluate, this will the llm.input_messages column in your exported data.

  • tool_call - information on the tool called and parameters included. If you've exported spans from Phoenix to evaluate, this will be the llm.function_call column in your exported data.

Parameter Extraction Only

This template instead evaluates only the parameter extraction step of a router:

You are comparing a function call response to a question and trying to determine if the generated call has extracted the exact right parameters from the question. Here is the data:
    [BEGIN DATA]
    ************
    [Question]: {question}
    ************
    [LLM Response]: {response}
    ************
    [END DATA]

Compare the parameters in the generated function against the JSON provided below.
The parameters extracted from the question must match the JSON below exactly.
Your response must be single word, either "correct", "incorrect", or "not-applicable",
and should not contain any text or characters aside from that word.

"correct" means the function call parameters match the JSON below and provides only relevant information.
"incorrect" means that the parameters in the function do not match the JSON schema below exactly, or the generated function does not correctly answer the user's question. You should also respond with "incorrect" if the response makes up information that is not in the JSON schema.
"not-applicable" means that response was not a function call.

Here is more information on each function:
{function_defintions}

When To Use Summarization Eval Template

This Eval helps evaluate the summarization results of a summarization task. The template variables are:

• document: The document text to summarize

• summary: The summary of the document

Summarization Eval Template

You are comparing the summary text and it's original document and trying to determine
if the summary is good. Here is the data:
    [BEGIN DATA]
    ************
    [Summary]: {output}
    ************
    [Original Document]: {input}
    [END DATA]
Compare the Summary above to the Original Document and determine if the Summary is
comprehensive, concise, coherent, and independent relative to the Original Document.
Your response must be a single word, either "good" or "bad", and should not contain any text
or characters aside from that. "bad" means that the Summary is not comprehensive,
concise, coherent, and independent relative to the Original Document. "good" means the
Summary is comprehensive, concise, coherent, and independent relative to the Original Document.

How To Run the Summarization Eval

import phoenix.evals.default_templates as templates
from phoenix.evals import (
    OpenAIModel,
    download_benchmark_dataset,
    llm_classify,
)

model = OpenAIModel(
    model_name="gpt-4",
    temperature=0.0,
)

#The rails is used to hold the output to specific values based on the template
#It will remove text such as ",,," or "..."
#Will ensure the binary value expected from the template is returned 
rails = list(templates.SUMMARIZATION_PROMPT_RAILS_MAP.values())
summarization_classifications = llm_classify(
    dataframe=df_sample,
    template=templates.SUMMARIZATION_PROMPT_TEMPLATE,
    model=model,
    rails=rails,
    provide_explanation=True, #optional to generate explanations for the value produced by the eval LLM
)

The above shows how to use the summarization Eval template.

Benchmark Results

This benchmark was obtained using notebook below. It was run using a Daily Mail CNN summarization dataset as a ground truth dataset. Each example in the dataset was evaluating using the SUMMARIZATION_PROMPT_TEMPLATE above, then the resulting labels were compared against the ground truth label in the summarization dataset to generate the confusion matrices below.

GPT-4 Results

When To Use Toxicity Eval Template

The following shows the results of the toxicity Eval on a toxic dataset test to identify if the AI response is racist, biased, or toxic. The template variables are:

• text: the text to be classified

Toxicity Eval Template

How To Run the Toxicity Eval

Benchmark Results

This benchmark was obtained using notebook below. It was run using the as a ground truth dataset. Each example in the dataset was evaluating using the TOXICITY_PROMPT_TEMPLATE above, then the resulting labels were compared against the ground truth label in the benchmark dataset to generate the confusion matrices below.

GPT-4 Results

Note: Palm is not useful for Toxicity detection as it always returns "" string for toxic inputs

When To Use Code Generation Eval Template

This Eval checks the correctness and readability of the code from a code generation process. The template variables are:

• query: The query is the coding question being asked

• code: The code is the code that was returned.

Code Generation Eval Template

How To Run the Code Generation Eval

The above shows how to use the code readability template.

Benchmark Results

This benchmark was obtained using notebook below. It was run using an as a ground truth dataset. Each example in the dataset was evaluating using the CODE_READABILITY_PROMPT_TEMPLATE above, then the resulting labels were compared against the ground truth label in the benchmark dataset to generate the confusion matrices below.

GPT-4 Results

Multimodal Templates

Multimodal evaluation templates enable users to evaluate tasks involving multiple input or output modalities, such as text, audio, or images. These templates provide a structured framework for constructing evaluation prompts, allowing LLMs to assess the quality, correctness, or relevance of outputs across diverse use cases.

The flexibility of multimodal templates makes them applicable to a wide range of scenarios, such as:

• Evaluating emotional tone in audio inputs, such as detecting user frustration or anger.

• Assessing the quality of image captioning tasks.

• Judging tasks that combine image and text inputs to produce contextualized outputs.

These examples illustrate how multimodal templates can be applied, but their versatility supports a broad spectrum of evaluation tasks tailored to specific user needs.

ClassificationTemplate

ClassificationTemplate is a class used to create evaluation prompts that are more complex than a simple string for classification tasks. We can also build prompts that consist of multiple message parts. We may include text, audio, or images in these messages, enabling us to construct multimodal evals if the LLM supports multimodal inputs.

By defining a ClassificationTemplate we can construct multi-part and multimodal evaluation templates by combining multiple PromptPartTemplateobjects.

• An evaluation prompt can consist of multiple PromptPartTemplate objects

• Each PromptPartTemplate can have a different content type

• Combine multiple PromptPartTemplate with templating variables to evaluate audio or image inputs

Structure of a ClassificationTemplate

A ClassificationTemplate consists of the following components:

1. Rails: These are the allowed classification labels for this evaluation task

2. Template: A list of PromptPartTemplate objects specifying the structure of the evaluation input. Each PromptPartTemplate includes:

   • content_type: The type of content (e.g., TEXT, AUDIO, IMAGE).

   • template: The string or object defining the content for that part.

3. Explanation_Template (optional): This is a separate structure used to generate explanations if explanations are enabled via llm_classify. If not enabled, this component is ignored.

Example: Intent Classification in Audio

The following example demonstrates how to create a ClassificationTemplate for an intent detection eval for a voice application:

Adapting to Different Modalities

The flexibility of ClassificationTemplate allows users to adapt it for various modalities, such as:

• Image Inputs: Replace PromptPartContentType.AUDIO with PromptPartContentType.IMAGE and update the templates accordingly.

• Mixed Modalities: Combine TEXT, AUDIO, and IMAGE for multimodal tasks requiring contextualized inputs.

Running the Evaluation with llm_classify

The llm_classify function can be used to run multimodal evaluations. This function supports input in the following formats:

• DataFrame: A DataFrame containing audio or image URLs, base64-encoded strings, and any additional data required for the evaluation.

• List: A collection of data items (e.g., audio or image URLs, list of base64 encoded strings).

Key Considerations for Input Data

• Public Links: If the data contains URLs for audio or image inputs, they must be publicly accessible for OpenAI to process them directly.

• Base64-Encoding: For private or local data, users must encode audio or image files as base64 strings and pass them to the function.

• Data Processor (optional): If links are not public and require transformation (e.g., base64 encoding), a data processor can be passed directly to llm_classify to handle the conversion in parallel, ensuring secure and efficient processing.

Using a Data Processor

A data processor enables efficient parallel processing of private or raw data into the required format.

Requirements

1. Consistent Input/Output: Input and output types should match, e.g., a series to a series for DataFrame processing.

2. Link Handling: Fetch data from provided links (e.g., cloud storage) and encode it in base64.

3. Column Consistency: The processed data must align with the columns referenced in the template.

Example: Processing Audio Links

The following is an example of a data processor that fetches audio from Google Cloud Storage, encodes it as base64, and assigns it to the appropriate column:

If your data is already base64-encoded, you can skip that step.

Performing the Evaluation

To run an evaluation, use the llm_classify function.

Before you begin:

You'll need two things to build your own LLM Eval:

1. A dataset to evaluate

2. A template prompt to use as the evaluation prompt on each row of data.

The dataset can have any columns you like, and the template can be structured however you like. The only requirement is that the dataset has all the columns your template uses.

We have two examples of templates below: CATEGORICAL_TEMPLATE and SCORE_TEMPLATE. The first must be used alongside a dataset with columns query and reference. The second must be used with a dataset that includes a column called context.

Feel free to set up your template however you'd like to match your dataset.

Preparing your data

You will need a dataset of results to evaluate. This dataset should be a pandas dataframe. If you are already collecting traces with Phoenix, you can export these traces and use them as the dataframe to evaluate:

trace_df = px.Client(endpoint="http://127.0.0.1:6006").get_spans_dataframe()

If your eval should have categorical outputs, use llm_classify.

If your eval should have numeric outputs, use llm_generate.

Categorical - llm_classify

The llm_classify function is designed for classification support both Binary and Multi-Class. The llm_classify function ensures that the output is clean and is either one of the "classes" or "UNPARSABLE"

A binary template looks like the following with only two values "irrelevant" and "relevant" that are expected from the LLM output:

CATEGORICAL_TEMPLATE = ''' You are comparing a reference text to a question and trying to determine if the reference text
contains information relevant to answering the question. Here is the data:
    [BEGIN DATA]
    ************
    [Question]: {query}
    ************
    [Reference text]: {reference}
    [END DATA]

Compare the Question above to the Reference text. You must determine whether the Reference text
contains information that can answer the Question. Please focus on whether the very specific
question can be answered by the information in the Reference text.
Your response must be single word, either "relevant" or "irrelevant",
and should not contain any text or characters aside from that word.
"irrelevant" means that the reference text does not contain an answer to the Question.
"relevant" means the reference text contains an answer to the Question. '''

The categorical template defines the expected output of the LLM, and the rails define the classes expected from the LLM:

• irrelevant

• relevant

from phoenix.evals import (
    llm_classify,
    OpenAIModel # see https://arize.com/docs/phoenix/evaluation/evaluation-models
    # for a full list of supported models
)

# The rails is used to hold the output to specific values based on the template
# It will remove text such as ",,," or "..."
# Will ensure the binary value expected from the template is returned
rails = ["irrelevant", "relevant"]
#MultiClass would be rails = ["irrelevant", "relevant", "semi-relevant"]
relevance_classifications = llm_classify(
    dataframe=<YOUR_DATAFRAME_GOES_HERE>,
    template=CATEGORICAL_TEMPLATE,
    model=OpenAIModel('gpt-4o', api_key=''),
    rails=rails
)

Snap to Rails Function

The classify uses a snap_to_rails function that searches the output string of the LLM for the classes in the classification list. It handles cases where no class is available, both classes are available or the string is a substring of the other class such as irrelevant and relevant.

#Rails examples
#Removes extra information and maps to class
llm_output_string = "The answer is relevant...!"
> "relevant"

#Removes "." and capitalization from LLM output and maps to class
llm_output_string = "Irrelevant."
>"irrelevant"

#No class in resposne
llm_output_string = "I am not sure!"
>"UNPARSABLE"

#Both classes in response
llm_output_string = "The answer is relevant i think, or maybe irrelevant...!"
>"UNPARSABLE"

A common use case is mapping the class to a 1 or 0 numeric value.

Numeric - llm_generate

The Phoenix library does support numeric score Evals if you would like to use them. A template for a score Eval looks like the following:

SCORE_TEMPLATE = """
You are a helpful AI bot that checks for grammatical, spelling and typing errors
in a document context. You are going to return a continous score for the
document based on the percent of grammatical and typing errors. The score should be
between 10 and 1. A score of 1 will be no grammatical errors in any word,
a score of 2 will be 20% of words have errors, a 5 score will be 50% errors,
a score of 7 is 70%, and a 10 score will be all words in the context have a
grammatical errors.

The following is the document context.

#CONTEXT
{context}
#ENDCONTEXT

#QUESTION
Please return a score between 10 and 1.
You will return no other text or language besides the score. Only return the score.
Please return in a format that is "the score is: 10" or "the score is: 1"
"""

We use the more generic llm_generate function that can be used for almost any complex eval that doesn't fit into the categorical type.

from phoenix.evals import (
    llm_generate,
    OpenAIModel # see https://arize.com/docs/phoenix/evaluation/evaluation-models
    # for a full list of supported models
)

test_results = llm_generate(
    dataframe=<YOUR_DATAFRAME_GOES_HERE>,
    template=SCORE_TEMPLATE,
    model=OpenAIModel(model='gpt-4o', api_key=''),
    verbose=True,
    # Callback function that will be called for each row of the dataframe
    output_parser=numeric_score_eval,
    # These two flags will add the prompt / response to the returned dataframe
    include_prompt=True,
    include_response=True,
)

def numeric_score_eval(output, row_index):
    # This is the function that will be called for each row of the 
    # dataframe after the eval is run
    row = df.iloc[row_index]
    score = self.find_score(output)

    return {"score": score}

def find_score(self, output):
    # Regular expression pattern
    # It looks for 'score is', followed by any characters (.*?), and then a float or integer
    pattern = r"score is.*?([+-]?(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?)"

    match = re.search(pattern, output, re.IGNORECASE)
    if match:
        # Extract and return the number
        return float(match.group(1))
    else:
        return None

The above is an example of how to run a score based Evaluation.

Logging Evaluations to Phoenix

from phoenix.trace import SpanEvaluations

px.Client().log_evaluations(
    SpanEvaluations(eval_name="Your Eval Display Name", dataframe=test_results)
)

Improving your Custom Eval

At this point, you've constructed a custom Eval, but you have no understanding of how accurate that Eval is. To test your eval, you can use the same techniques that you use to iterate and improve on your application.

1. Start with a labeled ground truth set of data. Each input would be a row of your dataframe of examples, and each labeled output would be the correct judge label

2. Test your eval on that labeled set of examples, and compare to the ground truth to calculate F1, precision, and recall scores. For an example of this, see Hallucinations

3. Tweak your prompt and retest. See https://github.com/Arize-ai/phoenix/blob/docs/docs/evaluation/how-to-evals/broken-reference/README.md for an example of how to do this in an automated way.

You can use cron to run evals client-side as your traces and spans are generated, augmenting your dataset with evaluations in an online manner. View the example in Github.

This example:

• Continuously queries a LangChain application to send new traces and spans to your Phoenix session

• Queries new spans once per minute and runs evals, including:

  • Hallucination

  • Q&A Correctness

  • Relevance

• Logs evaluations back to Phoenix so they appear in the UI

The evaluation script is run as a cron job, enabling you to adjust the frequency of the evaluation job:

* * * * * /path/to/python /path/to/run_evals.py

The above script can be run periodically to augment Evals in Phoenix.

Supported LLM Providers

We currently support the following LLM providers under phoenix.evals:

LLM Wrappers

OpenAIModel

To authenticate with OpenAI you will need, at a minimum, an API key. The model class will look for it in your environment, or you can pass it via argument as shown above. In addition, you can choose the specific name of the model you want to use and its configuration parameters. The default values specified above are common default values from OpenAI. Quickly instantiate your model as follows:

Azure OpenAI

The code snippet below shows how to initialize OpenAIModel for Azure:

Azure OpenAI supports specific options:

For full details on Azure OpenAI, check out the

Find more about the functionality available in our EvalModels in the section.

VertexAI

To authenticate with VertexAI, you must pass either your credentials or a project, location pair. In the following example, we quickly instantiate the VertexAI model as follows:

GeminiModel

Similar to VertexAIModel above for authentication

AnthropicModel

BedrockModel

To Authenticate, the following code is used to instantiate a session and the session is used with Phoenix Evals

MistralAIModel

Need to install extra dependency mistralai

LiteLLMModel

Need to install the extra dependency litellm>=1.0.3

You can choose among supported by LiteLLM. Make sure you have set the right environment variables set prior to initializing the model. For additional information about the environment variables for specific model providers visit:

Here is an example of how to initialize LiteLLMModel for llama3 using ollama.

Usage

In this section, we will showcase the methods and properties that our EvalModels have. First, instantiate your model from the. Once you've instantiated your model, you can get responses from the LLM by simply calling the model and passing a text string.

phoenix.evals.run_evals

Evaluates a pandas dataframe using a set of user-specified evaluators that assess each row for relevance of retrieved documents, hallucinations, toxicity, etc. Outputs a list of dataframes, one for each evaluator, that contain the labels, scores, and optional explanations from the corresponding evaluator applied to the input dataframe.

Parameters

• dataframe (pandas.DataFrame): A pandas dataframe in which each row represents an individual record to be evaluated. Each evaluator uses an LLM and an evaluation prompt template to assess the rows of the dataframe, and those template variables must appear as column names in the dataframe.

• evaluators (List[LLMEvaluator]): A list of evaluators to apply to the input dataframe. Each evaluator class accepts a as input, which is used in conjunction with an evaluation prompt template to evaluate the rows of the input dataframe and to output labels, scores, and optional explanations. Currently supported evaluators include:

  • HallucinationEvaluator: Evaluates whether a response (stored under an "output" column) is a hallucination given a query (stored under an "input" column) and one or more retrieved documents (stored under a "reference" column).

  • RelevanceEvaluator: Evaluates whether a retrieved document (stored under a "reference" column) is relevant or irrelevant to the corresponding query (stored under an "input" column).

  • ToxicityEvaluator: Evaluates whether a string (stored under an "input" column) contains racist, sexist, chauvinistic, biased, or otherwise toxic content.

  • QAEvaluator: Evaluates whether a response (stored under an "output" column) is correct or incorrect given a query (stored under an "input" column) and one or more retrieved documents (stored under a "reference" column).

  • SummarizationEvaluator: Evaluates whether a summary (stored under an "output" column) provides an accurate synopsis of an input document (stored under an "input" column).

  • SQLEvaluator: Evaluates whether a generated SQL query (stored under the "query_gen" column) and a response (stored under the "response" column) appropriately answer a question (stored under the "question" column).

• provide_explanation (bool, optional): If true, each output dataframe will contain an explanation column containing the LLM's reasoning for each evaluation.

• use_function_calling_if_available (bool, optional): If true, function calling is used (if available) as a means to constrain the LLM outputs. With function calling, the LLM is instructed to provide its response as a structured JSON object, which is easier to parse.

• verbose (bool, optional): If true, prints detailed information such as model invocation parameters, retries on failed requests, etc.

• concurrency (int, optional): The number of concurrent workers if async submission is possible. If not provided, a recommended default concurrency is set on a per-model basis.

Returns

• List[pandas.DataFrame]: A list of dataframes, one for each evaluator, all of which have the same number of rows as the input dataframe.

Usage

To use run_evals, you must first wrangle your LLM application data into a pandas dataframe either manually or by querying and exporting the spans collected by your Phoenix session. Once your dataframe is wrangled into the appropriate format, you can instantiate your evaluators by passing the model to be used during evaluation.

Run your evaluations by passing your dataframe and your list of desired evaluators.

Assuming your dataframe contains the "input", "reference", and "output" columns required by HallucinationEvaluator and QAEvaluator, your output dataframes should contain the results of the corresponding evaluator applied to the input dataframe, including columns for labels (e.g., "factual" or "hallucinated"), scores (e.g., 0 for factual labels, 1 for hallucinated labels), and explanations. If your dataframe was exported from your Phoenix session, you can then ingest the evaluations using phoenix.log_evaluations so that the evals will be visible as annotations inside Phoenix.

For an end-to-end example, see the .

phoenix.evals.PromptTemplate

Class used to store and format prompt templates.

Parameters

• text (str): The raw prompt text used as a template.

• delimiters (List[str]): List of characters used to locate the variables within the prompt template text. Defaults to ["{", "}"].

Attributes

• text (str): The raw prompt text used as a template.

• variables (List[str]): The names of the variables that, once their values are substituted into the template, create the prompt text. These variable names are automatically detected from the template text using the delimiters passed when initializing the class (see Usage section below).

Usage

Define a PromptTemplate by passing a text string and the delimiters to use to locate the variables. The default delimiters are { and }.

If the prompt template variables have been correctly located, you can access them as follows:

The PromptTemplate class can also understand any combination of delimiters. Following the example above, but getting creative with our delimiters:

Once you have a PromptTemplate class instantiated, you can make use of its format method to construct the prompt text resulting from substituting values into the variables. To do so, a dictionary mapping the variable names to the values is passed:

Note that once you initialize the PromptTemplate class, you don't need to worry about delimiters anymore, it will be handled for you.

phoenix.evals.llm_classify

Classifies each input row of the dataframe using an LLM. Returns a pandas.DataFrame where the first column is named label and contains the classification labels. An optional column named explanation is added when provide_explanation=True.

Parameters

• dataframe (pandas.DataFrame): A pandas dataframe in which each row represents a record to be classified. All template variable names must appear as column names in the dataframe (extra columns unrelated to the template are permitted).

• template (ClassificationTemplate, or str): The prompt template as either an instance of PromptTemplate or a string. If the latter, the variable names should be surrounded by curly braces so that a call to .format can be made to substitute variable values.

• model (BaseEvalModel): An LLM model class instance

• rails (List[str]): A list of strings representing the possible output classes of the model's predictions.

• system_instruction (Optional[str]): An optional system message for modals that support it

• verbose (bool, optional): If True, prints detailed info to stdout such as model invocation parameters and details about retries and snapping to rails. Default False.

• use_function_calling_if_available (bool, default=True): If True, use function calling (if available) as a means to constrain the LLM outputs. With function calling, the LLM is instructed to provide its response as a structured JSON object, which is easier to parse.

• provide_explanation (bool, default=False): If True, provides an explanation for each classification label. A column named explanation is added to the output dataframe. Note that this will default to using function calling if available. If the model supplied does not support function calling, llm_classify will need a prompt template that prompts for an explanation. For phoenix's pre-tested eval templates, the template is swapped out for a based template that prompts for an explanation.

Returns

• pandas.DataFrame: A dataframe where the label column (at column position 0) contains the classification labels. If provide_explanation=True, then an additional column named explanation is added to contain the explanation for each label. The dataframe has the same length and index as the input dataframe. The classification label values are from the entries in the rails argument or "NOT_PARSABLE" if the model's output could not be parsed.

phoenix.evals.llm_generate

Generates a text using a template using an LLM. This function is useful if you want to generate synthetic data, such as irrelevant responses

Parameters

• dataframe (pandas.DataFrame): A pandas dataframe in which each row represents a record to be used as in input to the template. All template variable names must appear as column names in the dataframe (extra columns unrelated to the template are permitted).

• template (Union[PromptTemplate, str]): The prompt template as either an instance of PromptTemplate or a string. If the latter, the variable names should be surrounded by curly braces so that a call to format can be made to substitute variable values.

• model (BaseEvalModel): An LLM model class.

• system_instruction (Optional[str], optional): An optional system message.

• output_parser (Callable[[str, int], Dict[str, Any]], optional): An optional function that takes each generated response and response index and parses it to a dictionary. The keys of the dictionary should correspond to the column names of the output dataframe. If None, the output dataframe will have a single column named "output". Default None.

Returns

• generations_dataframe (pandas.DataFrame): A dataframe where each row represents the generated output

Usage

Below we show how you can use llm_generate to use an llm to generate synthetic data. In this example, we use the llm_generate function to generate the capitals of countries but llm_generate can be used to generate any type of data such as synthetic questions, irrelevant responses, and so on.

llm_generate also supports an output parser so you can use this to generate data in a structured format. For example, if you want to generate data in JSON format, you ca prompt for a JSON object and then parse the output using the json library.