Skip to main content
An eval is an automated check on your agent’s output. The judge can be an LLM scoring against criteria you wrote, deterministic code, an agent harness that reads your traces at run time, or an HTTP endpoint you host. Whichever you use, it returns some combination of a label (e.g. correct, incorrect), a numeric score, and an explanation. Results attach to the data you scored and behave like any other attribute: filter traces by eval results, chart them in a dashboard, or set a monitor that fires when a label starts trending the wrong way. Evals run in two modes: continuously against live production traces, or on demand against a dataset or experiment. See Run evals on your data to learn more.
Arize AX traces view with summary charts for traffic, span latency, tokens and cost, and a custom eval metric, plus a traces table showing Span Evaluations tags per row alongside latency and token columns

Eval labels attached to your traces in Arize AX

​
Why evaluate

AI agents fail in ways traditional software doesn’t: confident falsehoods, plausible reasoning that lands on the wrong answer, the wrong context retrieved, the wrong tool called, a prompt tweak that quietly degrades quality for users whose traces you never read. None of it surfaces as an error. Your agent returns a 200, the trace looks fine, and the user gets a bad answer. Evals are how you catch it.

​
Human review vs. evals

Human review and evals do different jobs, and you need both. The two work together. Read traces by hand to learn how your agent fails, then encode each failure mode as an eval. Reviewing a sample by hand afterwards tells you whether the judge agrees with you, which is what makes its scores trustworthy on the traces you never read. See Align to human review.
Two parallel improvement cycles, agent improvement on the left and evaluator improvement on the right, sharing center nodes for collecting failure cases and annotating a golden dataset, then looping back to feed each cycle's next iteration

The two improvement cycles - agent on the left, evaluator on the right - sharing the golden dataset in the middle.

​
Where evals live

Evaluators are stored in the Eval Hub, under Evaluators in the left nav. The Hub is what makes an evaluator reusable rather than a one-off:
  • Define once, reuse anywhere - attach the same evaluator to online monitoring, offline batch runs, or dataset experiments without rewriting the prompt.
  • Versioned - every change tracked with commit messages.
  • Portable - map template variables to each data source’s columns, so one evaluator works across projects, datasets, and experiments with different shapes.
A second tab, Running Tasks, is where tasks execute those evaluators against your data. Building an evaluator and running it are deliberately separate: see Build evals, then Run evals on your data.
Evaluators page with Evaluator Hub tab selected, showing a table of LLM evaluators with scope, judge model, maintainer, and usage

Eval Hub lists your saved evaluators and their configuration

​
Next

Human review

Find failures by hand so you know what’s worth automating.

Build evals

Compare the four evaluator types and build one.

Run evals on your data

Attach an evaluator to live traces or a dataset.

Results and costs

Read results, chart them, control judge spend.