Skip to main content
The evaluator defines what to measure; the task defines which data, how much of it, and how often.

​
What is a task

A task connects an evaluator to a data source. Because evaluators live in the Eval Hub independently of any task, you build one judge and point it at as many places as you need - different projects, datasets, or experiments. Results attach automatically wherever that data lives. Every task sets four things:
Workflow diagram from create evaluator in Eval Hub through create task with target and sampling, runs over tracing or experiment data with scope, view results with scores and task logs, and investigate with view evals or jump to trace, with a loop back to edit or improve the evaluator

Evaluator and task workflow

You’ll need data flowing into Arize AX and an LLM provider configured first. See AI Provider Integrations.
Let Alyx set up the task. Press Cmd+L (macOS) or Ctrl+L (Windows/Linux) to open Alyx and try: “Create an online task that runs this eval on 10% of traces”

​
Column mapping

An evaluator’s variables stay unresolved until a task supplies them, which is what lets the same evaluator score data with different schemas. Adding an evaluator to a task opens a mapping row per variable:
  • The variable name as the row label
  • A dropdown to pick the span attribute (e.g., attributes.output.value, attributes.input.value) or dataset column that feeds it, populated from the columns present in your project or dataset
  • A free-text entry for attribute names the dropdown does not list
  • A live preview under each row showing the value from a sample record, so you can confirm the mapping before saving
Auto-mapping: When a variable name exactly matches a column name in your data source, Arize AX fills the mapping in for you. You can override it at any time.
For a code evaluator, the variables are the named arguments of its evaluate() method: at run time Arize AX reads the mapped attributes and passes them in as keyword arguments. For trace and session evaluators, a variable can also specify which span in the pattern supplies it. See Subquery-aware variable mapping.

​
Test evaluator code locally

While you can write and test evaluator code in the UI, it is often easier to iterate in a local Python environment. The Test in Code button generates starter code that you can run locally.
1

Click "Test in Code"

In the evaluator form, click the Test in Code button. The system validates your code first - if there are errors, they are shown before proceeding.
2

Copy the generated code

A slide-over opens with a generated Python notebook containing:
  • Environment setup (pip install arize)
  • Imports and data loading code using your actual API keys
  • Your evaluator class and a helper run_evaluators() function
3

Run locally

Paste the code into a Python script or Jupyter notebook and run it against your data. Once you are seeing the desired results, copy the updated evaluator class back into the UI.

​
Online or offline

The same evaluator runs in either mode. Only the target changes. Run both: offline evals gate what ships, online evals watch what shipped. Most teams start with a one-time backfill to set a baseline, then switch on a continuous task.

Online evals

Attach an evaluator to live traces.

Offline evals

Attach an evaluator to a dataset or experiment.
For the architectural split between platform-managed and self-managed evaluation, see Online vs offline evaluators.

​
After it runs

Results attach to whatever the task scored - spans for a project task, experiment rows for a dataset task. See Results and costs for where to read them and how to keep judge spend down.