> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluating RAG Retrieval Quality and Correctness
> Trace, evaluate, diagnose, and improve a retrieval-augmented generation (RAG) application in Arize AX, from retrieval relevance to chunking experiments.
## Overview
In this tutorial, you'll build a retrieval-augmented generation (RAG) application with [LlamaIndex](/docs/ax/integrations/python-agent-frameworks/llamaindex/llamaindex-tracing), trace it in Arize AX, then evaluate, diagnose, and improve its retrieval quality. The application is a simple question-answering chatbot that answers from a single sample document.
This tutorial is intended for AI engineers building RAG systems. It assumes you have basic knowledge of:
* Python and running notebooks
* How RAG works at a high level (retrieval + generation)
* LLM application concepts (embeddings, vector stores, prompts)
By the end of this tutorial, you'll be able to:
* Instrument a RAG application and send traces to Arize AX
* Evaluate retrieval relevance and answer correctness with LLM-as-a-judge
* Diagnose the three common retrieval failure modes using embeddings
* Run experiments to improve retrieval across chunk size, overlap, and `k`
## Background: how RAG retrieval works and where it fails
RAG connects your own data to an LLM. A user asks a question, an embedding is generated from the query, the most relevant context in your knowledge base is retrieved, and that context is added to the prompt sent to the LLM.
When a RAG application returns a bad answer, it usually traces back to one of **three retrieval failure modes**:
1. **Bad response.** The end answer is wrong or unhelpful, often surfaced by negative user feedback or low eval scores. This is usually a symptom of one of the failures below.
2. **Missing context.** The retriever couldn't find any documents close enough to the query, meaning users are asking about topics missing from your corpus.
3. **Most similar is not most relevant.** A document had the closest embedding to the query but wasn't actually the most relevant one to answer it.
For a walkthrough of these concepts, watch the video:
## Before you start
Before you begin, you should have:
* An Arize AX account. [Sign up here](https://app.arize.com/auth/join) and get your [space ID and API key](/docs/ax/get-started/get-started-tracing)
* An OpenAI API key
* An [LLM provider configured in Arize AX](/docs/ax/security-and-settings/integrations-playground/overview), required to run no-code evaluations in Step 2
* The required packages installed:
```bash theme={null}
pip install "arize>=8.0.0" arize-otel openinference-instrumentation-llama-index llama-index arize-phoenix-evals
```
## Step 1: Trace your RAG app
Arize AX auto-instruments many RAG stacks, including [LlamaIndex](/docs/ax/integrations/python-agent-frameworks/llamaindex/llamaindex-tracing) and [LangChain](/docs/ax/integrations/python-agent-frameworks/langchain/langchain-tracing), and supports [manual instrumentation](/docs/ax/instrument/manual-instrumentation) for custom ones. This tutorial uses LlamaIndex.
First, set your API keys as environment variables. Every step reads credentials from here, so run this once at the start:
```python theme={null}
import os
from getpass import getpass
os.environ["ARIZE_SPACE_ID"] = getpass("Enter your Arize space ID")
os.environ["ARIZE_API_KEY"] = getpass("Enter your Arize API key")
os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key")
```
Set up tracing with [arize-otel](https://pypi.org/project/arize-otel/), a convenience package that configures OpenTelemetry alongside [OpenInference](https://github.com/Arize-ai/openinference) auto-instrumentation, which maps LLM metadata to a standardized set of trace and span attributes. This sends every LlamaIndex call to Arize AX:
```python theme={null}
from arize.otel import register
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
tracer_provider = register(
space_id=os.environ["ARIZE_SPACE_ID"],
api_key=os.environ["ARIZE_API_KEY"],
project_name="rag-cookbook",
)
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)
```
Next, give the chatbot a knowledge base to answer from. We'll use an essay by Paul Graham, the sample document from [LlamaIndex's starter tutorial](https://docs.llamaindex.ai/en/stable/getting_started/starter_example/). Download it into a local `data/` folder:
```bash theme={null}
!mkdir data
!wget "https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt" -O data/paul_graham_essay.txt
```
Load the document, build a vector index over it, and expose that index as a query engine. This query engine is the RAG application you'll trace and evaluate:
```python theme={null}
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI
# Load the sample document
documents = SimpleDirectoryReader("data").load_data()
# Create Vector Store Index
index = VectorStoreIndex.from_documents(documents)
# Turn it into a query engine using gpt-4.1-mini
query_engine = index.as_query_engine(llm=OpenAI(model="gpt-4.1-mini"))
# Create a query
response = query_engine.query("What did Paul Graham work on?")
print(response)
```
This creates a trace in Arize AX, which visualizes inputs, outputs, retrieved chunks, embeddings, and LLM prompt parameters.
Send a few more queries so you have several traces to evaluate in the next step:
```python theme={null}
for question in [
"What did Paul Graham do growing up?",
"What did Paul Graham study in college?",
"What did Paul Graham do at Y Combinator?",
"Why did Paul Graham start painting?",
]:
query_engine.query(question)
```
## Step 2: Evaluate retrieval quality and correctness
Now that traces are flowing into Arize AX, evaluate them so you don't have to manually inspect every trace. We'll score two things:
* **Retrieval relevance:** is the retrieved context relevant to the question? This is judged on the **retriever span**, which carries both the query (`input`) and the retrieved documents.
* **Answer correctness:** is the final answer correct given the retrieved context? This draws on two spans: the query and context come from the **retriever span**, and the answer comes from the **LLM span**.
Because the inputs to these evaluators live on different spans, retrieval relevance is a **span-level** eval on retriever spans, while correctness is a **trace-level** eval that reads across the retriever and LLM spans in a trace.
Arize AX uses the [Phoenix Evals](https://arize.com/docs/phoenix/evaluation) library for LLM-as-a-judge, and its [prebuilt templates](/docs/ax/evaluate/evaluators/llm-as-a-judge) (including RAG relevance, QA correctness, and hallucination) are available both in the UI and in code.
### Option A: Run a no-code evaluation task (recommended)
Create a task that runs an evaluator from the Eval Hub directly on your traces, with no export and no dataframe. You can do this from the UI, from [Alyx](/docs/ax/alyx), or the `ax` CLI. In the UI, click **New Task** from the **Evaluators** page, select your `rag-cookbook` project as the data source, then **Add Evaluator** and pick a prebuilt template from the Eval Hub.
**Retrieval relevance (span-level).** Add the prebuilt **Retrieval Relevance** template, scope it to **span** level, and filter to retriever spans with `attributes.openinference.span.kind = 'RETRIEVER'`. Both variables come from that one span:
| Variable | Column |
| ------------- | -------------------------------- |
| `{input}` | `input` |
| `{reference}` | `attributes.retrieval.documents` |
**Answer correctness (trace-level).** Correctness needs the question, the retrieved context, and the answer, which live on different spans. Add the prebuilt **QA Correctness** template, scope it to **trace** level, and use a [multi-span query](/docs/ax/evaluate/run-evals-on-traces#multi-span-queries-trace-and-session-evals) so each variable reads from the right span role:
| Subquery | Filter |
| -------- | ------------------------------------------------------------------- |
| **A** | `attributes.openinference.span.kind = 'RETRIEVER'` |
| **B** | `attributes.openinference.span.kind = 'LLM' AND status_code = 'OK'` |
Expression `A => B` (the retriever is the direct parent of the LLM span), with this variable mapping:
| Variable | Column | Subquery |
| ------------ | -------------------------------- | -------- |
| `{question}` | `input` | A |
| `{context}` | `attributes.retrieval.documents` | A |
| `{answer}` | `output` | B |
For each task, set the cadence (one-time backfill or run continuously), sampling rate, and any filters, then click **Create Task**. Results attach automatically to your spans in the Tracing view. For the full task workflow, sampling guidance, and multi-span queries, see [Run online evals on traces](/docs/ax/evaluate/run-evals-on-traces).
### Option B: Run evaluations in code
Use this when you need to run evals on large datasets, incorporate external data, or want full control over execution and cost. Export your spans, run evals with Phoenix Evals, and log the results back onto your spans.
First, export your spans and split them by role:
```python theme={null}
import os
from datetime import datetime, timedelta
from arize import ArizeClient
client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
end_time = datetime.now()
start_time = end_time - timedelta(days=1)
spans_df = client.spans.export_to_df(
space_id=os.environ["ARIZE_SPACE_ID"],
project_name="rag-cookbook",
start_time=start_time,
end_time=end_time,
)
# Retriever spans carry the query and the retrieved documents
retriever_df = spans_df[
spans_df["attributes.openinference.span.kind"] == "RETRIEVER"
].copy()
```
Define the judge and a retrieval-relevance evaluator, then run it. The template placeholders (`{input}`, `{reference}`) are filled from same-named columns in each row, so map the span attributes onto those column names first:
```python theme={null}
from phoenix.evals import LLM, create_classifier, evaluate_dataframe
retriever_df["input"] = retriever_df["attributes.input.value"]
retriever_df["reference"] = retriever_df["attributes.retrieval.documents"]
llm = LLM(provider="openai", model="gpt-4.1")
RELEVANCE_TEMPLATE = """You are comparing a reference text to a question and trying to determine
if the reference text contains information relevant to answering the question. Here is the data:
[BEGIN DATA]
************
[Question]: {input}
************
[Reference text]: {reference}
[END DATA]
Is the Reference text relevant to answering the Question? Answer "relevant" or "unrelated"."""
relevance_evaluator = create_classifier(
name="retrieval_relevance",
llm=llm,
prompt_template=RELEVANCE_TEMPLATE,
choices={"relevant": 1.0, "unrelated": 0.0},
)
results_df = evaluate_dataframe(
dataframe=retriever_df,
evaluators=[relevance_evaluator],
)
```
Then log the results back onto your spans. `to_annotation_dataframe` explodes the scores into `label`, `score`, and `explanation` columns and carries the `context.span_id` through; [`update_evaluations`](/docs/api-clients/python/version-8/client-resources/spans#update-evaluations) expects those under the `eval..*` convention:
```python theme={null}
from phoenix.evals.utils import to_annotation_dataframe
annotation_df = to_annotation_dataframe(results_df, ["retrieval_relevance"])
eval_df = annotation_df.rename(columns={
"label": "eval.retrieval_relevance.label",
"score": "eval.retrieval_relevance.score",
"explanation": "eval.retrieval_relevance.explanation",
})[["context.span_id",
"eval.retrieval_relevance.label",
"eval.retrieval_relevance.score",
"eval.retrieval_relevance.explanation"]]
client.spans.update_evaluations(
space_id=os.environ["ARIZE_SPACE_ID"],
project_name="rag-cookbook",
dataframe=eval_df,
force_http=True,
)
```
To also score **answer correctness** in code, join the retrieved context onto the LLM (or root) span by `context.trace_id` so each row has the question, context, and answer together, then run a correctness classifier with a `{question}` / `{context}` / `{answer}` template and log it the same way. For the full column conventions and the 14-day span window, see [Run online evals on traces](/docs/ax/evaluate/run-evals-on-traces#create-a-task).
## Step 3: Diagnose retrieval failures with embeddings
With evals in place, use Arize AX to trace bad responses back to their root cause. These embedding-based diagnostics are most useful at production scale, comparing your live queries against your full document corpus. Here's how Arize AX surfaces each of the three failure modes introduced earlier.
**Issue 1: Bad response.** Arize AX automatically surfaces clusters of traces with low scores, using the retrieval relevance and correctness evals you logged in Step 2. You can also add [user feedback on spans](/docs/ax/observe/take-action/annotate-traces) as another signal. A bad response is usually a symptom of one of the retrieval failures below.
**Issue 2: Missing context.** The retriever couldn't find documents close enough to the query, meaning users ask about topics missing from your corpus. Visualize query density to find topics you need to add documentation for: set your production data as user queries and your corpus as the baseline, then look for clusters of query embeddings with no nearby context embeddings.
**Issue 3: Most similar is not most relevant.** A retrieved document had the closest embedding but wasn't the most relevant. Arize AX uncovers this with LLM-assisted relevance ranking (precision\@k), sending the query and retrieved context to an LLM to score relevance.
## Step 4: Improve retrieval with experiments
Once you can measure quality, experiment to improve it. Generate a benchmark question set, then compare retrieval settings (chunk size, chunk overlap, and the number of documents retrieved, `k`) as [experiments](/docs/ax/develop/datasets-and-experiments) in Arize AX.
First, generate a synthetic set of test questions from the essay and put them in a dataframe with an `input` column:
```python theme={null}
import pandas as pd
from llama_index.llms.openai import OpenAI
GEN_TEMPLATE = """
You are an assistant that generates Q&A questions about the content below.
The questions should involve specific facts and figures, names, and elements of the text.
Do not ask any questions where the answer is not in the content.
Respond with one question per line. Do not include numbering. Generate 10 unique questions.
[START CONTENT]
{content}
[END CONTENT]
"""
with open("data/paul_graham_essay.txt") as f:
essay = f.read()
resp = OpenAI(model="gpt-4.1").complete(GEN_TEMPLATE.format(content=essay))
questions = [q for q in resp.text.strip().split("\n") if q.strip()]
questions_df = pd.DataFrame(questions, columns=["input"])
```
Define a helper that builds a query engine for a given configuration, then wrap it as an experiment **task**. The task answers one question per dataset row and returns the answer along with the retrieved context. The context is carried through only so the evaluators can score it.
```python theme={null}
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.node_parser import SentenceSplitter
documents = SimpleDirectoryReader("data").load_data()
def build_query_engine(chunk_size, chunk_overlap, k):
splitter = SentenceSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
index = VectorStoreIndex.from_documents(documents, transformations=[splitter])
return index.as_query_engine(similarity_top_k=k, llm=OpenAI(model="gpt-4.1-mini"))
def make_task(chunk_size, chunk_overlap, k):
engine = build_query_engine(chunk_size, chunk_overlap, k)
def task(dataset_row):
response = engine.query(dataset_row["input"])
return {
"output": str(response),
"reference": "\n".join(n.text for n in response.source_nodes),
}
return task
```
Define two Phoenix Evals classifiers and wrap each as an experiment **evaluator** that returns an `EvaluationResult` with a label, score, and explanation. An evaluator receives the task's `output` and the `dataset_row`, so it can read the question, retrieved context, and answer:
```python theme={null}
from phoenix.evals import LLM, create_classifier
from arize.experiments import EvaluationResult
llm = LLM(provider="openai", model="gpt-4.1")
RELEVANCE_TEMPLATE = """You are comparing a reference text to a question and trying to determine
if the reference text contains information relevant to answering the question. Here is the data:
[BEGIN DATA]
************
[Question]: {input}
************
[Reference text]: {reference}
[END DATA]
Is the Reference text relevant to answering the Question? Answer "relevant" or "unrelated"."""
CORRECTNESS_TEMPLATE = """You are given a question, an answer, and reference text. Determine whether
the answer correctly answers the question based on the reference text. Here is the data:
[BEGIN DATA]
************
[Question]: {input}
************
[Reference]: {reference}
************
[Answer]: {output}
[END DATA]
Answer "correct" or "incorrect"."""
relevance_classifier = create_classifier(
name="relevance", llm=llm, prompt_template=RELEVANCE_TEMPLATE,
choices={"relevant": 1.0, "unrelated": 0.0},
)
correctness_classifier = create_classifier(
name="correctness", llm=llm, prompt_template=CORRECTNESS_TEMPLATE,
choices={"correct": 1.0, "incorrect": 0.0},
)
def relevance(output, dataset_row):
score = relevance_classifier.evaluate({
"input": dataset_row["input"],
"reference": output["reference"],
})[0]
return EvaluationResult(score=score.score, label=score.label, explanation=score.explanation)
def correctness(output, dataset_row):
score = correctness_classifier.evaluate({
"input": dataset_row["input"],
"reference": output["reference"],
"output": output["output"],
})[0]
return EvaluationResult(score=score.score, label=score.label, explanation=score.explanation)
```
Create a dataset from the questions, then run one experiment per configuration. `client.experiments.run` executes the task across every example, applies the evaluators, and logs the results to Arize AX so you can compare how chunk size, overlap, and `k` affect retrieval quality in the UI:
```python theme={null}
import os
from arize import ArizeClient
client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
space_id = os.environ["ARIZE_SPACE_ID"]
dataset_name = "rag-experiments"
client.datasets.create(space=space_id, name=dataset_name, examples=questions_df)
def run_experiment(name, chunk_size, chunk_overlap, k):
client.experiments.run(
name=name,
dataset=dataset_name,
space=space_id,
task=make_task(chunk_size, chunk_overlap, k),
evaluators=[relevance, correctness],
)
# Compare retrieval settings
run_experiment("k2-chunk200", chunk_size=200, chunk_overlap=10, k=2)
run_experiment("k4-chunk200", chunk_size=200, chunk_overlap=10, k=4)
run_experiment("k2-chunk1000", chunk_size=1000, chunk_overlap=50, k=2)
```
For the full datasets and experiments API, see [Run an experiment](/docs/ax/develop/datasets-and-experiments/run-experiment) and [Log an experiment](/docs/ax/develop/datasets-and-experiments/log-experiment).
## Summary
In this tutorial, you built a RAG application and used Arize AX to:
* Trace retrieval, embeddings, and LLM calls end to end
* Score retrieval relevance and answer correctness with LLM-as-a-judge
* Diagnose missing-context and irrelevant-retrieval failures using embeddings
* Run experiments to improve retrieval across chunk size, overlap, and `k`
## Next steps
* Manually create [retriever spans](/docs/ax/instrument/manual-instrumentation) that log your retriever inputs and outputs
* Evaluate across more dimensions such as hallucination, citation, and user frustration with [prebuilt evaluators](/docs/ax/evaluate/create-evaluators#tutorial-run-pre-built-evals-on-your-traces)
* Go deeper on [datasets and experiments](/docs/ax/develop/datasets-and-experiments)
When a RAG application returns a bad answer, it usually traces back to one of **three retrieval failure modes**:
1. **Bad response.** The end answer is wrong or unhelpful, often surfaced by negative user feedback or low eval scores. This is usually a symptom of one of the failures below.
2. **Missing context.** The retriever couldn't find any documents close enough to the query, meaning users are asking about topics missing from your corpus.
3. **Most similar is not most relevant.** A document had the closest embedding to the query but wasn't actually the most relevant one to answer it.
For a walkthrough of these concepts, watch the video:
## Before you start
Before you begin, you should have:
* An Arize AX account. [Sign up here](https://app.arize.com/auth/join) and get your [space ID and API key](/docs/ax/get-started/get-started-tracing)
* An OpenAI API key
* An [LLM provider configured in Arize AX](/docs/ax/security-and-settings/integrations-playground/overview), required to run no-code evaluations in Step 2
* The required packages installed:
```bash theme={null}
pip install "arize>=8.0.0" arize-otel openinference-instrumentation-llama-index llama-index arize-phoenix-evals
```
## Step 1: Trace your RAG app
Arize AX auto-instruments many RAG stacks, including [LlamaIndex](/docs/ax/integrations/python-agent-frameworks/llamaindex/llamaindex-tracing) and [LangChain](/docs/ax/integrations/python-agent-frameworks/langchain/langchain-tracing), and supports [manual instrumentation](/docs/ax/instrument/manual-instrumentation) for custom ones. This tutorial uses LlamaIndex.
First, set your API keys as environment variables. Every step reads credentials from here, so run this once at the start:
```python theme={null}
import os
from getpass import getpass
os.environ["ARIZE_SPACE_ID"] = getpass("Enter your Arize space ID")
os.environ["ARIZE_API_KEY"] = getpass("Enter your Arize API key")
os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key")
```
Set up tracing with [arize-otel](https://pypi.org/project/arize-otel/), a convenience package that configures OpenTelemetry alongside [OpenInference](https://github.com/Arize-ai/openinference) auto-instrumentation, which maps LLM metadata to a standardized set of trace and span attributes. This sends every LlamaIndex call to Arize AX:
```python theme={null}
from arize.otel import register
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
tracer_provider = register(
space_id=os.environ["ARIZE_SPACE_ID"],
api_key=os.environ["ARIZE_API_KEY"],
project_name="rag-cookbook",
)
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)
```
Next, give the chatbot a knowledge base to answer from. We'll use an essay by Paul Graham, the sample document from [LlamaIndex's starter tutorial](https://docs.llamaindex.ai/en/stable/getting_started/starter_example/). Download it into a local `data/` folder:
```bash theme={null}
!mkdir data
!wget "https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt" -O data/paul_graham_essay.txt
```
Load the document, build a vector index over it, and expose that index as a query engine. This query engine is the RAG application you'll trace and evaluate:
```python theme={null}
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI
# Load the sample document
documents = SimpleDirectoryReader("data").load_data()
# Create Vector Store Index
index = VectorStoreIndex.from_documents(documents)
# Turn it into a query engine using gpt-4.1-mini
query_engine = index.as_query_engine(llm=OpenAI(model="gpt-4.1-mini"))
# Create a query
response = query_engine.query("What did Paul Graham work on?")
print(response)
```
This creates a trace in Arize AX, which visualizes inputs, outputs, retrieved chunks, embeddings, and LLM prompt parameters.
Send a few more queries so you have several traces to evaluate in the next step:
```python theme={null}
for question in [
"What did Paul Graham do growing up?",
"What did Paul Graham study in college?",
"What did Paul Graham do at Y Combinator?",
"Why did Paul Graham start painting?",
]:
query_engine.query(question)
```
## Step 2: Evaluate retrieval quality and correctness
Now that traces are flowing into Arize AX, evaluate them so you don't have to manually inspect every trace. We'll score two things:
* **Retrieval relevance:** is the retrieved context relevant to the question? This is judged on the **retriever span**, which carries both the query (`input`) and the retrieved documents.
* **Answer correctness:** is the final answer correct given the retrieved context? This draws on two spans: the query and context come from the **retriever span**, and the answer comes from the **LLM span**.
Because the inputs to these evaluators live on different spans, retrieval relevance is a **span-level** eval on retriever spans, while correctness is a **trace-level** eval that reads across the retriever and LLM spans in a trace.
Arize AX uses the [Phoenix Evals](https://arize.com/docs/phoenix/evaluation) library for LLM-as-a-judge, and its [prebuilt templates](/docs/ax/evaluate/evaluators/llm-as-a-judge) (including RAG relevance, QA correctness, and hallucination) are available both in the UI and in code.
### Option A: Run a no-code evaluation task (recommended)
Create a task that runs an evaluator from the Eval Hub directly on your traces, with no export and no dataframe. You can do this from the UI, from [Alyx](/docs/ax/alyx), or the `ax` CLI. In the UI, click **New Task** from the **Evaluators** page, select your `rag-cookbook` project as the data source, then **Add Evaluator** and pick a prebuilt template from the Eval Hub.
**Retrieval relevance (span-level).** Add the prebuilt **Retrieval Relevance** template, scope it to **span** level, and filter to retriever spans with `attributes.openinference.span.kind = 'RETRIEVER'`. Both variables come from that one span:
| Variable | Column |
| ------------- | -------------------------------- |
| `{input}` | `input` |
| `{reference}` | `attributes.retrieval.documents` |
**Answer correctness (trace-level).** Correctness needs the question, the retrieved context, and the answer, which live on different spans. Add the prebuilt **QA Correctness** template, scope it to **trace** level, and use a [multi-span query](/docs/ax/evaluate/run-evals-on-traces#multi-span-queries-trace-and-session-evals) so each variable reads from the right span role:
| Subquery | Filter |
| -------- | ------------------------------------------------------------------- |
| **A** | `attributes.openinference.span.kind = 'RETRIEVER'` |
| **B** | `attributes.openinference.span.kind = 'LLM' AND status_code = 'OK'` |
Expression `A => B` (the retriever is the direct parent of the LLM span), with this variable mapping:
| Variable | Column | Subquery |
| ------------ | -------------------------------- | -------- |
| `{question}` | `input` | A |
| `{context}` | `attributes.retrieval.documents` | A |
| `{answer}` | `output` | B |
For each task, set the cadence (one-time backfill or run continuously), sampling rate, and any filters, then click **Create Task**. Results attach automatically to your spans in the Tracing view. For the full task workflow, sampling guidance, and multi-span queries, see [Run online evals on traces](/docs/ax/evaluate/run-evals-on-traces).
### Option B: Run evaluations in code
Use this when you need to run evals on large datasets, incorporate external data, or want full control over execution and cost. Export your spans, run evals with Phoenix Evals, and log the results back onto your spans.
First, export your spans and split them by role:
```python theme={null}
import os
from datetime import datetime, timedelta
from arize import ArizeClient
client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
end_time = datetime.now()
start_time = end_time - timedelta(days=1)
spans_df = client.spans.export_to_df(
space_id=os.environ["ARIZE_SPACE_ID"],
project_name="rag-cookbook",
start_time=start_time,
end_time=end_time,
)
# Retriever spans carry the query and the retrieved documents
retriever_df = spans_df[
spans_df["attributes.openinference.span.kind"] == "RETRIEVER"
].copy()
```
Define the judge and a retrieval-relevance evaluator, then run it. The template placeholders (`{input}`, `{reference}`) are filled from same-named columns in each row, so map the span attributes onto those column names first:
```python theme={null}
from phoenix.evals import LLM, create_classifier, evaluate_dataframe
retriever_df["input"] = retriever_df["attributes.input.value"]
retriever_df["reference"] = retriever_df["attributes.retrieval.documents"]
llm = LLM(provider="openai", model="gpt-4.1")
RELEVANCE_TEMPLATE = """You are comparing a reference text to a question and trying to determine
if the reference text contains information relevant to answering the question. Here is the data:
[BEGIN DATA]
************
[Question]: {input}
************
[Reference text]: {reference}
[END DATA]
Is the Reference text relevant to answering the Question? Answer "relevant" or "unrelated"."""
relevance_evaluator = create_classifier(
name="retrieval_relevance",
llm=llm,
prompt_template=RELEVANCE_TEMPLATE,
choices={"relevant": 1.0, "unrelated": 0.0},
)
results_df = evaluate_dataframe(
dataframe=retriever_df,
evaluators=[relevance_evaluator],
)
```
Then log the results back onto your spans. `to_annotation_dataframe` explodes the scores into `label`, `score`, and `explanation` columns and carries the `context.span_id` through; [`update_evaluations`](/docs/api-clients/python/version-8/client-resources/spans#update-evaluations) expects those under the `eval.
**Issue 3: Most similar is not most relevant.** A retrieved document had the closest embedding but wasn't the most relevant. Arize AX uncovers this with LLM-assisted relevance ranking (precision\@k), sending the query and retrieved context to an LLM to score relevance.
## Step 4: Improve retrieval with experiments
Once you can measure quality, experiment to improve it. Generate a benchmark question set, then compare retrieval settings (chunk size, chunk overlap, and the number of documents retrieved, `k`) as [experiments](/docs/ax/develop/datasets-and-experiments) in Arize AX.
First, generate a synthetic set of test questions from the essay and put them in a dataframe with an `input` column:
```python theme={null}
import pandas as pd
from llama_index.llms.openai import OpenAI
GEN_TEMPLATE = """
You are an assistant that generates Q&A questions about the content below.
The questions should involve specific facts and figures, names, and elements of the text.
Do not ask any questions where the answer is not in the content.
Respond with one question per line. Do not include numbering. Generate 10 unique questions.
[START CONTENT]
{content}
[END CONTENT]
"""
with open("data/paul_graham_essay.txt") as f:
essay = f.read()
resp = OpenAI(model="gpt-4.1").complete(GEN_TEMPLATE.format(content=essay))
questions = [q for q in resp.text.strip().split("\n") if q.strip()]
questions_df = pd.DataFrame(questions, columns=["input"])
```
Define a helper that builds a query engine for a given configuration, then wrap it as an experiment **task**. The task answers one question per dataset row and returns the answer along with the retrieved context. The context is carried through only so the evaluators can score it.
```python theme={null}
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.node_parser import SentenceSplitter
documents = SimpleDirectoryReader("data").load_data()
def build_query_engine(chunk_size, chunk_overlap, k):
splitter = SentenceSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
index = VectorStoreIndex.from_documents(documents, transformations=[splitter])
return index.as_query_engine(similarity_top_k=k, llm=OpenAI(model="gpt-4.1-mini"))
def make_task(chunk_size, chunk_overlap, k):
engine = build_query_engine(chunk_size, chunk_overlap, k)
def task(dataset_row):
response = engine.query(dataset_row["input"])
return {
"output": str(response),
"reference": "\n".join(n.text for n in response.source_nodes),
}
return task
```
Define two Phoenix Evals classifiers and wrap each as an experiment **evaluator** that returns an `EvaluationResult` with a label, score, and explanation. An evaluator receives the task's `output` and the `dataset_row`, so it can read the question, retrieved context, and answer:
```python theme={null}
from phoenix.evals import LLM, create_classifier
from arize.experiments import EvaluationResult
llm = LLM(provider="openai", model="gpt-4.1")
RELEVANCE_TEMPLATE = """You are comparing a reference text to a question and trying to determine
if the reference text contains information relevant to answering the question. Here is the data:
[BEGIN DATA]
************
[Question]: {input}
************
[Reference text]: {reference}
[END DATA]
Is the Reference text relevant to answering the Question? Answer "relevant" or "unrelated"."""
CORRECTNESS_TEMPLATE = """You are given a question, an answer, and reference text. Determine whether
the answer correctly answers the question based on the reference text. Here is the data:
[BEGIN DATA]
************
[Question]: {input}
************
[Reference]: {reference}
************
[Answer]: {output}
[END DATA]
Answer "correct" or "incorrect"."""
relevance_classifier = create_classifier(
name="relevance", llm=llm, prompt_template=RELEVANCE_TEMPLATE,
choices={"relevant": 1.0, "unrelated": 0.0},
)
correctness_classifier = create_classifier(
name="correctness", llm=llm, prompt_template=CORRECTNESS_TEMPLATE,
choices={"correct": 1.0, "incorrect": 0.0},
)
def relevance(output, dataset_row):
score = relevance_classifier.evaluate({
"input": dataset_row["input"],
"reference": output["reference"],
})[0]
return EvaluationResult(score=score.score, label=score.label, explanation=score.explanation)
def correctness(output, dataset_row):
score = correctness_classifier.evaluate({
"input": dataset_row["input"],
"reference": output["reference"],
"output": output["output"],
})[0]
return EvaluationResult(score=score.score, label=score.label, explanation=score.explanation)
```
Create a dataset from the questions, then run one experiment per configuration. `client.experiments.run` executes the task across every example, applies the evaluators, and logs the results to Arize AX so you can compare how chunk size, overlap, and `k` affect retrieval quality in the UI:
```python theme={null}
import os
from arize import ArizeClient
client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
space_id = os.environ["ARIZE_SPACE_ID"]
dataset_name = "rag-experiments"
client.datasets.create(space=space_id, name=dataset_name, examples=questions_df)
def run_experiment(name, chunk_size, chunk_overlap, k):
client.experiments.run(
name=name,
dataset=dataset_name,
space=space_id,
task=make_task(chunk_size, chunk_overlap, k),
evaluators=[relevance, correctness],
)
# Compare retrieval settings
run_experiment("k2-chunk200", chunk_size=200, chunk_overlap=10, k=2)
run_experiment("k4-chunk200", chunk_size=200, chunk_overlap=10, k=4)
run_experiment("k2-chunk1000", chunk_size=1000, chunk_overlap=50, k=2)
```
For the full datasets and experiments API, see [Run an experiment](/docs/ax/develop/datasets-and-experiments/run-experiment) and [Log an experiment](/docs/ax/develop/datasets-and-experiments/log-experiment).
## Summary
In this tutorial, you built a RAG application and used Arize AX to:
* Trace retrieval, embeddings, and LLM calls end to end
* Score retrieval relevance and answer correctness with LLM-as-a-judge
* Diagnose missing-context and irrelevant-retrieval failures using embeddings
* Run experiments to improve retrieval across chunk size, overlap, and `k`
## Next steps
* Manually create [retriever spans](/docs/ax/instrument/manual-instrumentation) that log your retriever inputs and outputs
* Evaluate across more dimensions such as hallucination, citation, and user frustration with [prebuilt evaluators](/docs/ax/evaluate/create-evaluators#tutorial-run-pre-built-evals-on-your-traces)
* Go deeper on [datasets and experiments](/docs/ax/develop/datasets-and-experiments)