Trace and session evals

​

When to use each scope

​

How trace and session evals work

​

Session-only variables

• {conversation} is the session transcript Arize AX builds as a JSON array with one input / output pair per turn, taken from each trace’s root span in chronological order. It needs no variable mapping, and it is valid only at session scope.
• {turn_data} is the string Arize AX produces by rendering a Turn Definition template you write once per turn, then joining the rendered turns with TURN 1, TURN 2, and so on as separators. Use it when the default input/output transcript leaves out something the judge needs, such as tool results or retrieved context.

​

How variables are filled in

​

Where results land

​

Set up a trace or session eval

• Trace eval
• Session eval

1

Open the New Evaluator form

Open Evaluators in the left navigation and click New Evaluator, or open a trace from the traces table and click Add Trace Eval in the trace header, an action that pre-sets the scope for you.

2

Name it and set Scope to Trace

In the left pane, enter a Name and set Scope to Trace. The Result Column Name is auto-populated from the name and becomes the trace_eval.<name> column.

3

Pick the judge model and write the template

Choose the judge model above the template, then write the template, referencing the variables the judge needs, for example {question}, {tool_calls}, and {output}. Because the eval sees the whole trace, write criteria about the path taken, not about a single response.

4

Define Choices and Optimization Direction

Below the template, define Choices and their scores, then set the Optimization Direction so Arize AX knows whether a higher or a lower score is better.

5

Pick a Data Source

In the Configuration column, choose Project and select the project whose traces you want scored. The picker also offers Dataset, but a dataset is scored one example at a time whatever the scope, so trace grouping applies only to a project.

6

Filter Traces

Narrow the traces the evaluator runs on. This step accepts a multi-span query, which combines several named span filters with operators such as AND and “followed by” to describe a pattern of spans within one trace. Multi-span queries are available only at trace and session scope.

7

Preview Matching Data

Read the sample the filter returns to confirm it admits the traces you expect before you map any variables.

8

Map Variable to Data

Point each template variable at a column. Arize AX joins the values from every matching span with commas, in chronological order, and Data preview under each mapping shows the value it pulled from the latest matching trace.

9

Set Task Configuration

Name the task, set the Sampling Rate (%), and turn on either Run Continuously to score new traces as they arrive or One-Time Backfill to score history once. Advanced holds an optional LLM Override and the Enable tracing toggle. See Run online evals on traces for how to pick a sampling rate.

10

Create the evaluator

Click Create Evaluator to save it to the Evaluator Hub for reuse, or Create and Run Evaluator to save it and start the task you just configured.

1

Open the New Evaluator form

Open Evaluators in the left navigation and click New Evaluator, or use the Add Session Eval action next to the session ID in the session header. Sessions exist only when your application sets session.id on its spans; see OpenInference context managers if that attribute is missing.

2

Name it and set Scope to Session

In the left pane, enter a Name and set Scope to Session. The Result Column Name is auto-populated from the name and becomes the session_eval.<name> column.

3

Choose Conversation or Turn data

Pick Conversation to have Arize AX build the transcript from each turn’s input and output. Picking it inserts {conversation} into your template. Pick Turn data when the judge needs more than input and output; that option inserts {turn_data} and reveals the Turn Definition field.

4

Write the Turn Definition (Turn data only)

Lay out one turn using {variable} references, for example:

User: {user_msg}
Assistant: {assistant_msg}
Tool result: {tool_result}

Arize AX renders this template once per turn and joins the results with TURN 1, TURN 2 markers. Variables you leave unmapped render empty rather than failing the turn.

5

Write the template, then define Choices and Optimization Direction

Choose the judge model, write the template around {conversation} or {turn_data}, then define Choices and their scores and set the Optimization Direction so Arize AX knows whether a higher or a lower score is better.

6

Pick a Data Source

In the Configuration column, choose Project and select the project whose sessions you want scored. The picker also offers Dataset, but a dataset is scored one example at a time whatever the scope, so session grouping applies only to a project.

7

Filter Sessions

Narrow the sessions the evaluator runs on, then read Preview Matching Data to confirm the filter admits the sessions you expect. This step accepts a multi-span query, which combines several named span filters with operators such as AND and “followed by” to describe a pattern of spans within one session.

8

Map Variable to Data and read the Data Preview

A {conversation} eval needs no mapping, since Arize AX assembles the transcript from the matched session spans. For a {turn_data} eval, map each Turn Definition variable to a column; Arize AX reads each variable only from the turn it belongs to, so turn 2’s values never appear in turn 1. The Data Preview shows the full prompt the judge will receive for the latest matching session, with every variable and every assembled turn filled in. Iterate on the Turn Definition and the mappings until the preview reads the way you want.

9

Set Task Configuration

Name the task, set the Sampling Rate (%), and turn on either Run Continuously to score new sessions as they arrive or One-Time Backfill to score history once. Advanced holds an optional LLM Override and the Enable tracing toggle.

10

Create the evaluator

Click Create Evaluator to save it to the Evaluator Hub for reuse, or Create and Run Evaluator to save it and start the task you just configured.

​

View results

• Trace detail: open a trace from the traces table and read the Trace Evals row in the header. Each result appears as a colored label, and hovering over one shows its score and explanation.
• Session detail: open a session from the Sessions tab and read the Session Evals row in the header, where the labels behave the same way.
• Traces table: add the Trace Evaluations or Session Evaluations column from the column picker to scan results across many units, and filter on trace_eval.<name>.label or session_eval.<name>.score to isolate failures.

​

Log trace and session evals from code

# Trace eval: one row per trace, keyed on that trace's root span id
trace_eval_df["context.span_id"] = trace_eval_df["root_span_id"]
trace_eval_df = trace_eval_df.rename(columns={
    "label": "trace_eval.trajectory.label",
    "score": "trace_eval.trajectory.score",
    "explanation": "trace_eval.trajectory.explanation",
})

# Session eval: one row per session, keyed on the root span of the session's first trace
session_eval_df["context.span_id"] = session_eval_df["first_root_span_id"]
session_eval_df = session_eval_df.rename(columns={
    "label": "session_eval.correctness.label",
    "score": "session_eval.correctness.score",
    "explanation": "session_eval.correctness.explanation",
})

client.spans.update_evaluations(
    space_id=os.environ["ARIZE_SPACE_ID"],
    project_name="your-project-name",
    dataframe=trace_eval_df,
)

client.spans.update_evaluations(
    space_id=os.environ["ARIZE_SPACE_ID"],
    project_name="your-project-name",
    dataframe=session_eval_df,
)

​

Further reading