New Evaluator

Choose a template or start from scratch

Custom Eval
LLM-As-A-JudgeAn LLM scores each record against your prompt template
Code EvaluatorScore records with custom Python code
Agent-as-a-JudgeRun evaluations in a Claude Code sandbox
Remote EvalScore spans with an external evaluation endpoint
New EvaluatorAlyx
Name
response_correctness
Type
LLM-As-A-Judge
Scope
Span
Model
A\ claude-fable-5-2
LLM Parameters
Template <>
Evaluate the response below. Input: {input} Response: {output} Answer with a single label and explain your reasoning.
Labels
correct
1
incorrect
0
Add Label
1Pick the spans to evaluate
Filter to the spans this eval should score.
Project
support-agent
attributes.openinference.span.kind equals LLM ×
Filter spans...
Matching resultsOct 18 - Nov 17
SpansColumns
KindNameInput
LLMclassify_ticketYou classify customer support tickets…
LLMclassify_ticketYou classify customer support tickets…
LLMclassify_ticketYou classify customer support tickets…
2Connect the eval to your spans
From a matching span, map each {variable} to its value, then give it a test run.
{input}
input
You classify customer support tickets. Label the ticket with exactly one category: billing, shipping or technical.

Ticket: The app logged me out and charged me again.
{output}
output
{"category": "billing"}
Test Evaluator On Spans
3Choose how the eval runs
Name your task, set how much data to score, and whether to run on new data, past data, or both.
Name
Nov 17, 3:16 PM
Sampling Rate
100
%
Run Continuously
One-Time Backfill
Name
valid_category
Scope
Span
Define Imports
Write all required import statements as if it were a python file View Available Packages
1from typing import Any, Optional
2import json
3from arize.experimental.datasets.experiments.evaluators.base import (
4 EvaluationResult,
5 CodeEvaluator,
6)
Define Code Evaluator Class
Write an evaluate() method that scores your data and returns an EvaluationResult. How to write a code evaluator
1class ValidCategory(CodeEvaluator):
2 """Checks for one known ticket category."""
3
4 CATEGORIES = ("billing", "shipping", "technical")
5
6 def evaluate(
7 self,
8 *,
9 output: Optional[str] = None,
10 **kwargs: Any,
11 ) -> EvaluationResult:
12 category = json.loads(output)["category"]
13 ok = category in self.CATEGORIES
14 return EvaluationResult(
15 label="valid" if ok else "invalid",
16 score=float(ok),
17 )
1Pick the spans to evaluate
Filter to the spans this eval should score.
Project
support-agent
attributes.openinference.span.kind equals LLM ×
Filter spans...
Matching resultsOct 18 - Nov 17
SpansColumns
KindNameInput
LLMclassify_ticketYou classify customer support tickets…
LLMclassify_ticketYou classify customer support tickets…
LLMclassify_ticketYou classify customer support tickets…
2Connect the eval to your spans
From a matching span, map each argument to its value, then give it a test run.
output
Select or search a column to map to...attributes.output.value
{"category": "billing"}
Test Code On Spans
3Choose how the eval runs
Name your task, set how much data to score, and whether to run on new data, past data, or both.
Name
Nov 17, 3:18 PM
Sampling Rate
100
%
Run Continuously
One-Time Backfill
Name
category_check
Scope
Span
Endpoint URL
https://evals.example.com/v1/evaluate
Arize will POST each span to this URL and expect a label and/or score in return. How to create a remote evaluator
Headers
Add Header
Input Schema
Declare the fields your endpoint reads under input. Each field there becomes a column mapping, auto-matched to the closest span attribute.
1{
2 "type": "object",
3 "required": ["metadata", "input"],
4 "properties": {
5 "metadata": {
6 "type": "object",
7 "description": "Auto-injected by Arize. Do not map these fields.",
8 "properties": {
9 "request_id": { "type": "string" },
10 "span_id": { "type": "string" }
11 }
12 },
13 "input": {
14 "type": "object",
15 "properties": {
16 "input": { "type": "string" },
17 "output": { "type": "string" }
18 }
19 }
20 }
21}
1Pick the spans to evaluate
Filter to the spans this eval should score.
Project
support-agent
attributes.openinference.span.kind equals LLM ×
Filter spans...
Matching resultsOct 18 - Nov 17
SpansColumns
KindNameInput
LLMclassify_ticketYou classify customer support tickets…
LLMclassify_ticketYou classify customer support tickets…
LLMclassify_ticketYou classify customer support tickets…
2Connect the eval to your spans
From a matching span, map each {variable} to its value, then give it a test run.
{input}
attributes.input.value
You classify customer support tickets. Label the ticket with exactly one category: billing, shipping or technical.

Ticket: The app logged me out and charged me again.
{output}
attributes.output.value
{"category": "billing"}
Test Remote On Spans
3Choose how the eval runs
Name your task, set how much data to score, and whether to run on new data, past data, or both.
Name
Nov 17, 3:18 PM
Sampling Rate
100
%
Run Continuously
One-Time Backfill
Name
tool_usage
How this works:
  1. Write what to evaluate in plain language
  2. The agent reads your traces at run time. No column mapping needed!
  3. Attach to a task to choose project and date range.
Harness
A\ Claude Code
Model
claude-fable-5-2
Scoring Instructions
Did the agent select and use tools appropriately? Check whether the right tools were called, whether any calls were unnecessary, and whether the final answer used the tool outputs correctly.
Labels
Let agent decide labels
Agent Skills Add a Skill
No skills attached. Skills give the evaluator access to external data (e.g. a GitHub repo with custom retrieval code) at run time.
1Pick a Data Source
Select a project to see options.
support-agent
2Choose how the eval runs
Name your task, set how much data to score, and whether to run on new data, past data, or both.
Name
tool usage check
Run Continuously
One-Time Backfill
Create EvaluatorCreate and Run Evaluator