> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Trace and session evals
> Score a full request or a full conversation with one evaluator. How Arize AX assembles data at trace and session scope, and how to set up each.
export const AskAlyx = ({children}) => {
const gradientId = `askAlyxGradient-${Math.random().toString(36).slice(2)}`;
return ;
};
A span eval scores one span. A trace eval scores every span in a single request, and a session eval scores every trace that shares a `session.id`. Each returns one result for the thing it scores, so a trace eval returns one label, score, and explanation for the whole request, and a session eval returns one label, score, and explanation for the whole conversation.
This guide covers how Arize AX assembles the data a trace or session evaluator sees, and how to configure a trace eval and a session eval. For the conceptual distinction between span, trace, and session, see [Evaluation levels](/docs/ax/concepts/evaluators/evaluation-levels).
## When to use each scope
| Scope | Unit evaluated | Questions it answers |
| :---------- | :--------------------------------- | :-------------------------------------------------------------------------------------------------- |
| **Span** | One span | Was the right tool selected? Was this retrieval relevant? Was this response correct? |
| **Trace** | Every span in one request | Was the sequence of tool calls correct? Did the agent loop? Did it skip a step? |
| **Session** | Every trace sharing a `session.id` | Did the agent keep context across turns? Did the user reach their goal? Did the conversation stall? |
Pick the narrowest scope that can answer your question. A session eval passes far more context to the judge than a span eval does, and that extra context raises cost and gives the judge more text to weigh.
Throughout this page, **unit** means whatever one evaluator run scores: a single trace for a trace eval, and a single session for a session eval.
## How trace and session evals work
Every evaluator has a scope, set with the **Scope** dropdown on the evaluator form, and that scope determines whether the evaluator scores a span, a trace, or a session. You run an evaluator by attaching it to a task, which connects it to a data source and defines what to score and how often. Every evaluator on one task must share the same scope, so a task is span-level, trace-level, or session-level, not a mix. To run evaluators at different scopes against the same project, create a separate task for each scope.
### Session-only variables
Session evaluators have two variables that Arize AX assembles for you rather than reading from a span attribute, one entry per turn, where a turn is one trace in the session. The two are mutually exclusive, and you choose between them with the **Conversation** and **Turn data** radio buttons on the evaluator form.
* **`{conversation}`** is the session transcript Arize AX builds as a JSON array with one `input` / `output` pair per turn, taken from each trace's root span in chronological order. It needs no variable mapping, and it is valid only at session scope.
* **`{turn_data}`** is the string Arize AX produces by rendering a **Turn Definition** template you write once per turn, then joining the rendered turns with `TURN 1`, `TURN 2`, and so on as separators. Use it when the default input/output transcript leaves out something the judge needs, such as tool results or retrieved context.
### How variables are filled in
At span scope, each template variable reads one value from one span. At trace and session scope, a variable can match many spans, so the matching values are concatenated with commas in chronological order. For trace evals, spans are ordered by span timestamp. For session evals, spans are grouped into traces first, and traces are ordered by earliest span timestamp.
Each value is truncated at 100,000 characters, and after that truncation a session's fully assembled value is capped at 100,000 characters. Some judge models raise both limits.
### Where results land
Each unit produces one result, written to that unit's root span. Results are stored under a prefixed column name so span, trace, and session evals with the same name never collide:
| Scope | Column prefix | Written to |
| :------ | :-------------------- | :----------------------------------------- |
| Span | `eval.` | The evaluated span |
| Trace | `trace_eval.` | The trace's root span |
| Session | `session_eval.` | The root span of the session's first trace |
Sampling on a trace or session task applies per unit, so 10% sampling on a session task evaluates 10% of sessions rather than 10% of spans.
When a task runs, Arize AX collects each unit's spans from a window that extends 24 hours before the time window the task itself evaluates, so a session that has been idle for longer than that contributes only its recent traces. Arize AX searches back 7 days to find a unit's root span, so results still land on the correct span for long-running sessions.
## Set up a trace or session eval
Trace and session evaluators are built on the same form, which has two halves. The left pane defines the evaluator itself: its name, scope, judge model, template, and choices. The numbered **Configuration** column on the right points that evaluator at data and sets up the task that runs it. The steps below follow the form in order. Session evaluators add one further decision about how each turn reaches the judge.
Ask [Alyx](/docs/ax/alyx), the Arize AX assistant, to build a trace-scoped evaluator for you: try *"Create a trace-scoped evaluator that checks whether the agent called tools in a sensible order."*
Open **Evaluators** in the left navigation and click **New Evaluator**, or open a trace from the traces table and click **Add Trace Eval** in the trace header, an action that pre-sets the scope for you.
In the left pane, enter a **Name** and set **Scope** to **Trace**. The **Result Column Name** is auto-populated from the name and becomes the `trace_eval.` column.
Choose the judge model above the template, then write the template, referencing the variables the judge needs, for example `{question}`, `{tool_calls}`, and `{output}`. Because the eval sees the whole trace, write criteria about the path taken, not about a single response.
Below the template, define **Choices** and their scores, then set the **Optimization Direction** so Arize AX knows whether a higher or a lower score is better.
In the **Configuration** column, choose **Project** and select the project whose traces you want scored. The picker also offers **Dataset**, but a dataset is scored one example at a time whatever the scope, so trace grouping applies only to a project.
Narrow the traces the evaluator runs on. This step accepts a [multi-span query](/docs/ax/observe/tracing/view-and-manage-traces#multi-span-filters), which combines several named span filters with operators such as `AND` and "followed by" to describe a pattern of spans within one trace. Multi-span queries are available only at trace and session scope.
Read the sample the filter returns to confirm it admits the traces you expect before you map any variables.
Point each template variable at a column. Arize AX joins the values from every matching span with commas, in chronological order, and **Data preview** under each mapping shows the value it pulled from the latest matching trace.
Name the task, set the **Sampling Rate (%)**, and turn on either **Run Continuously** to score new traces as they arrive or **One-Time Backfill** to score history once. **Advanced** holds an optional **LLM Override** and the **Enable tracing** toggle. See [Run online evals on traces](/docs/ax/evaluate/run-evals-on-traces) for how to pick a sampling rate.
Click **Create Evaluator** to save it to the [Evaluator Hub](/docs/ax/evaluate/create-evaluators#evaluator-hub) for reuse, or **Create and Run Evaluator** to save it and start the task you just configured.
Ask [Alyx](/docs/ax/alyx), the Arize AX assistant, to build a session-scoped evaluator for you: try *"Create an evaluator that checks whether the agent maintained context across the whole session."*
Open **Evaluators** in the left navigation and click **New Evaluator**, or use the **Add Session Eval** action next to the session ID in the session header. Sessions exist only when your application sets `session.id` on its spans; see [OpenInference context managers](/docs/ax/concepts/otel-openinference/context-managers) if that attribute is missing.
In the left pane, enter a **Name** and set **Scope** to **Session**. The **Result Column Name** is auto-populated from the name and becomes the `session_eval.` column.
Pick **Conversation** to have Arize AX build the transcript from each turn's input and output. Picking it inserts `{conversation}` into your template. Pick **Turn data** when the judge needs more than input and output; that option inserts `{turn_data}` and reveals the **Turn Definition** field.
Lay out one turn using `{variable}` references, for example:
```
User: {user_msg}
Assistant: {assistant_msg}
Tool result: {tool_result}
```
Arize AX renders this template once per turn and joins the results with `TURN 1`, `TURN 2` markers. Variables you leave unmapped render empty rather than failing the turn.
Choose the judge model, write the template around `{conversation}` or `{turn_data}`, then define **Choices** and their scores and set the **Optimization Direction** so Arize AX knows whether a higher or a lower score is better.
In the **Configuration** column, choose **Project** and select the project whose sessions you want scored. The picker also offers **Dataset**, but a dataset is scored one example at a time whatever the scope, so session grouping applies only to a project.
Narrow the sessions the evaluator runs on, then read **Preview Matching Data** to confirm the filter admits the sessions you expect. This step accepts a [multi-span query](/docs/ax/observe/tracing/view-and-manage-traces#multi-span-filters), which combines several named span filters with operators such as `AND` and "followed by" to describe a pattern of spans within one session.
A `{conversation}` eval needs no mapping, since Arize AX assembles the transcript from the matched session spans. For a `{turn_data}` eval, map each Turn Definition variable to a column; Arize AX reads each variable only from the turn it belongs to, so turn 2's values never appear in turn 1. The **Data Preview** shows the full prompt the judge will receive for the latest matching session, with every variable and every assembled turn filled in. Iterate on the Turn Definition and the mappings until the preview reads the way you want.
Name the task, set the **Sampling Rate (%)**, and turn on either **Run Continuously** to score new sessions as they arrive or **One-Time Backfill** to score history once. **Advanced** holds an optional **LLM Override** and the **Enable tracing** toggle.
Click **Create Evaluator** to save it to the [Evaluator Hub](/docs/ax/evaluate/create-evaluators#evaluator-hub) for reuse, or **Create and Run Evaluator** to save it and start the task you just configured.
The steps above describe the **New Evaluator** form. Attaching an evaluator that already exists in the [Evaluator Hub](/docs/ax/evaluate/create-evaluators#evaluator-hub) opens the same Configuration column under the title **Use Evaluator(s) – Task Configuration**, where the primary button reads **Run Evaluator(s)** and the Configuration steps are numbered from the point you enter.
**Test Evaluator On Spans** is disabled for trace- and session-scoped evaluators when the data source is a project, so use the data preview under **Map Variable to Data** to confirm what the judge receives before you run the task.
## View results
Trace and session results appear alongside your traces once the task runs.
* **Trace detail:** open a trace from the traces table and read the **Trace Evals** row in the header. Each result appears as a colored label, and hovering over one shows its score and explanation.
* **Session detail:** open a session from the **Sessions** tab and read the **Session Evals** row in the header, where the labels behave the same way.
* **Traces table:** add the **Trace Evaluations** or **Session Evaluations** column from the column picker to scan results across many units, and filter on `trace_eval..label` or `session_eval..score` to isolate failures.
## Log trace and session evals from code
If you run evals in your own environment, log results back with the prefix that matches the scope. Every result needs a `context.span_id` column, and for trace and session evals that column holds the id of the unit's root span, which is the trace's root span for a trace eval and the root span of the session's first trace for a session eval.
```python theme={null}
# Trace eval: one row per trace, keyed on that trace's root span id
trace_eval_df["context.span_id"] = trace_eval_df["root_span_id"]
trace_eval_df = trace_eval_df.rename(columns={
"label": "trace_eval.trajectory.label",
"score": "trace_eval.trajectory.score",
"explanation": "trace_eval.trajectory.explanation",
})
# Session eval: one row per session, keyed on the root span of the session's first trace
session_eval_df["context.span_id"] = session_eval_df["first_root_span_id"]
session_eval_df = session_eval_df.rename(columns={
"label": "session_eval.correctness.label",
"score": "session_eval.correctness.score",
"explanation": "session_eval.correctness.explanation",
})
client.spans.update_evaluations(
space_id=os.environ["ARIZE_SPACE_ID"],
project_name="your-project-name",
dataframe=trace_eval_df,
)
client.spans.update_evaluations(
space_id=os.environ["ARIZE_SPACE_ID"],
project_name="your-project-name",
dataframe=session_eval_df,
)
```
For the full export, evaluate, and log flow, including how far back evals can be applied, see [Run online evals on traces](/docs/ax/evaluate/run-evals-on-traces#create-a-task).
## Further reading
Task filters, sampling rates, cadence, and run logs in full.
The canonical trace eval: judging the path an agent took.
A worked trace-scoped example, end to end.
A worked session-scoped example across a multi-turn conversation.