Evaluation health for AI agents

Your evals need
evaluating, too.

Find missing coverage, broken evals, and checks that cost more than they need to. Review proposed fixes before applying them.

Get started free Book a demo

Eval Health

Active travel-agent/booking-api

Healthy evaluators21No open insight against these evaluators

Coverage gaps3Missing signals the agent found

Efficiency2Would be better as code

Effectiveness3Not providing meaningful signal

Needs review8Waiting for Accept or Reject

Issues found8 findings from recent runs3 findings from recent runs2 findings from recent runs3 findings from recent runs8 findings from recent runs

Coverage×Efficiency×Effectiveness×Needs review×

CoverageNov 8, 4:01 PM

#1 Response toxicity has no eval on this project

1,240 matching spansReview needed

EffectivenessNov 8, 11:12 AM

#2 Hallucination eval gives no signal

100 matching spansReview needed

EfficiencyNov 7, 6:30 PM

#3 Exact-match check is running as an LLM judge

412 matching spansReview needed

CoverageNov 7, 9:04 AM

#4 User frustration is unscored at session level

142 matching sessionsReview needed

EfficiencyNov 6, 2:18 PM

#5 Valid JSON runs as an LLM judge

389 matching spansReview needed

EffectivenessNov 5, 10:41 AM

#6 Tool selection eval has an unmapped variable

318 matching spansReview needed

CoverageNov 4, 5:55 PM

#7 Booking completion is never evaluated

61 matching sessionsReview needed

EffectivenessNov 4, 8:20 AM

#8 Itinerary quality eval runs two versions at once

274 matching spansReview needed

Response toxicity has no eval on this project

CoverageMute✓ Mark as fixed

response-toxicity · Found on Nov 2, 9:27 AM · Last seen Nov 8, 4:01 PM

Suggested actions

Create evalCreate eval "Response Toxicity" and run it on this project▾ View details

Dismiss✓ Accept and run

Overview

The project has no toxicity eval, so none of its 1,240 user-facing LLM spans are judged for harmful content before it reaches a traveler.

Plan

Eval — adapted from the toxicity product template. Labels: toxic=1, not_toxic=0. Direction: minimize. Granularity: span.

Exact-match check is running as an LLM judge

EfficiencyMute✓ Mark as fixed

exact-match-v2 · Found on Nov 3, 7:12 AM · Last seen Nov 7, 6:30 PM

Suggested actions

Convert to code evalReplace the LLM judge on "Exact Match" with a code eval▾ View details

Dismiss✓ Accept and run

Overview

This evaluator checks whether the answer matches a fixed string but calls an LLM every time. Code returned the same label on all 412 sampled spans.

Plan

Code eval — compares output.value against expected.value. Labels: match=1, no_match=0. Spend after the change: $0, from $1.30 per 1k runs.

Hallucination eval gives no signal

EffectivenessMute✓ Mark as fixed

hallucination-v1 · Found on Oct 26, 3:40 PM · Last seen Nov 8, 11:12 AM

Suggested actions

Detach evaluatorDetach "hallucination-v1" from this project▾ View details

Dismiss✓ Accept and run

Overview

hallucination-v1 returned the same label on all 100 sampled spans, so it never separates a good answer from a bad one on this project.

Plan

Detach — stop the evaluator and the $8.40 per 1k runs it costs. Re-attach once its prompt separates factual from hallucinated on a labeled set.

Review evals
Find gaps and inefficiencies
Review proposed fixes

An eval can run perfectly and tell you nothing.

Missing coverage

Response toxicity, 1,240 spans
1. Evalnone attachedunscored
Important behaviors go unevaluated.
Unreliable judgments

Hallucination eval
1. Evalsame label, every spanno decision
Evaluators produce inconsistent or uninformative results.
Wasted spend

Valid JSON
1. LLMjudgecode could check this
Expensive or redundant evaluations consume resources without adding useful signal.

Find coverage gaps, broken evals, and wasted spend.

Findings3 open

CoverageNo eval checks whether cancellation requests succeed.
EffectivenessAn evaluator reads the wrong response field.
EfficiencyAn LLM judge checks for valid JSON.

Add a task-completion evalIllustrative example

Trace evidence

User request: “Cancel my order.”
Agent action: update_shipping
Agent response: “Your shipping address has been updated.”

What the agent foundCurrent evals don’t check whether the requested task was completed.

Proposed changeCreate an evaluator that checks task completion and run it on this project.

AcceptDismissAwaiting review

Eval Health

Keep your evals useful.

Coverage

Find missing eval coverage.

Find behaviors your evals miss.

Response toxicity has no eval on this project

CoverageMute✓ Mark as fixed

response-toxicity · Found on Nov 2, 9:27 AM · Last seen Nov 8, 4:01 PM

Suggested actions

Create evalCreate eval "Response Toxicity" and run it on this project▴ Hide details

Dismiss✓ Accept and run

Model

claude-haiku-5

Prompt template

14 lines

You are evaluating an AI assistant's reply for harmful content.

    [Assistant answer]: {output}

Show all 14 lines

Labels

toxic, not_toxic

Overview

The project has no toxicity eval, so none of its 1,240 user-facing LLM spans are judged for harmful content before it reaches a traveler.

Plan

Eval — adapted from the toxicity product template. Labels: toxic=1, not_toxic=0. Direction: minimize. Granularity: span.

Effectiveness

Review fixes for unreliable evals.

Review evaluator and configuration fixes.

Hallucination eval gives no signal

EffectivenessMute✓ Mark as fixed

hallucination-v1 · Found on Oct 26, 3:40 PM · Last seen Nov 8, 11:12 AM

Suggested actions

Detach evaluatorDetach "hallucination-v1" from this project▴ Hide details

Dismiss✓ Accept and run

Model

claude-opus-5

Prompt template

12 lines

You are checking whether the answer is supported by the context.

    [Context]: {context}

Show all 12 lines

Labels

hallucinated, factual

Overview

hallucination-v1 returned the same label on all 100 sampled spans, so it never separates a good answer from a bad one on this project.

Plan

Detach — stop the evaluator and the $8.40 per 1k runs it costs. Re-attach once its prompt separates factual from hallucinated on a labeled set.

Efficiency

Reduce unnecessary eval costs.

Find checks that could use code or a cheaper model.

Exact-match check is running as an LLM judge

EfficiencyMute✓ Mark as fixed

exact-match-v2 · Found on Nov 3, 7:12 AM · Last seen Nov 7, 6:30 PM

Suggested actions

Convert to code evalReplace the LLM judge on "Exact Match" with a code eval▴ Hide details

Dismiss✓ Accept and run

Language

Python

Code

9 lines

def exact_match(output, expected):
    a, b = output.get("value", ""), expected.get("value", "")
    return {"label": "match" if a.strip() == b.strip() else "no_match"}

Show all 9 lines

Labels

match, no_match

Overview

This evaluator checks whether the answer matches a fixed string but calls an LLM every time. Code returned the same label on all 412 sampled spans.

Plan

Code eval — compares output.value against expected.value. Labels: match=1, no_match=0. Spend after the change: $0, from $1.30 per 1k runs.

Response toxicity has no eval on this project

CoverageMute✓ Mark as fixed

response-toxicity · Found on Nov 2, 9:27 AM · Last seen Nov 8, 4:01 PM

Suggested actions

Create evalCreate eval "Response Toxicity" and run it on this project▴ Hide details

Dismiss✓ Accept and run

Model

claude-haiku-5

Prompt template

14 lines

You are evaluating an AI assistant's reply for harmful content.

    [Assistant answer]: {output}

Show all 14 lines

Labels

toxic, not_toxic

Overview

The project has no toxicity eval, so none of its 1,240 user-facing LLM spans are judged for harmful content before it reaches a traveler.

Plan

Eval — adapted from the toxicity product template. Labels: toxic=1, not_toxic=0. Direction: minimize. Granularity: span.

Hallucination eval gives no signal

EffectivenessMute✓ Mark as fixed

hallucination-v1 · Found on Oct 26, 3:40 PM · Last seen Nov 8, 11:12 AM

Suggested actions

Detach evaluatorDetach "hallucination-v1" from this project▴ Hide details

Dismiss✓ Accept and run

Model

claude-opus-5

Prompt template

12 lines

You are checking whether the answer is supported by the context.

    [Context]: {context}

Show all 12 lines

Labels

hallucinated, factual

Overview

hallucination-v1 returned the same label on all 100 sampled spans, so it never separates a good answer from a bad one on this project.

Plan

Detach — stop the evaluator and the $8.40 per 1k runs it costs. Re-attach once its prompt separates factual from hallucinated on a labeled set.

Exact-match check is running as an LLM judge

EfficiencyMute✓ Mark as fixed

exact-match-v2 · Found on Nov 3, 7:12 AM · Last seen Nov 7, 6:30 PM

Suggested actions

Convert to code evalReplace the LLM judge on "Exact Match" with a code eval▴ Hide details

Dismiss✓ Accept and run

Language

Python

Code

9 lines

def exact_match(output, expected):
    a, b = output.get("value", ""), expected.get("value", "")
    return {"label": "match" if a.strip() == b.strip() else "no_match"}

Show all 9 lines

Labels

match, no_match

Overview

This evaluator checks whether the answer matches a fixed string but calls an LLM every time. Code returned the same label on all 412 sampled spans.

Plan

Code eval — compares output.value against expected.value. Labels: match=1, no_match=0. Spend after the change: $0, from $1.30 per 1k runs.

Book a demo

Arize AX

Managed Agents

Automate recurring AI engineering work.

Eval Health is an Arize managed agent.

Start from a ready-made agent for the job, or build your own with access to your traces, code, and tools.

Signal

Find and investigate agent failures.

Reviews production traces on a schedule, groups recurring failures into ranked issues, and proposes fixes for review.

Learn more

Cost Agent

Find ways to reduce AI costs.

Finds unnecessary spend in model choices, repeated calls, and context, then recommends changes with estimated savings.

Learn more

Templates

Start from a template, or build your own.

Managed Agents include starting points for debugging, incident response, cost analysis, and recurring health checks.

Pick one and adapt it to your workflow, or start from scratch.

Connect the agent to the traces, code, and tools it needs for the task.

HarnessesClaude CodeCodexCursorOpenCodeHermesOpenClaw

SandboxesArizeClaude Managed AgentsVercel SandboxDaytona

SkillsGitHubArizeDynatraceDatadogCustom skill

Agent StudioAlyx

Describe a task, then choose an agent config to reuse a saved setup, or configure the managed agent manually.

Configuration

Harness*Claude CodeThe harness that executes your task

Sandbox*ArizeThe compute environment for the session

AI Provider*Arize managed (default)

Agent roleManaged Agent (Default)The permissions the run gets, separate from the harness and the skills

RepoSelect a repository

Skills+ Add skill

Task*

Describe the task — e.g. “Trace the checkout agent and flag any tool-call failures from the last 24h.”

Set up as automation

Book a demo

Give your evals a health check.

Keep your evaluation system useful as your AI applications evolve.

Book a demo

Docs

Start here

Build your playbook

See it in production

Company

Your evals needevaluating, too.

Eval Health

Response toxicity has no eval on this project

Overview

Plan

Exact-match check is running as an LLM judge

Overview

Plan

Hallucination eval gives no signal

Overview

Plan

Review evals

Find gaps and inefficiencies

Review proposed fixes

An eval can run perfectly and tell you nothing.

Find coverage gaps, broken evals, and wasted spend.

Keep your evals useful.

Find missing eval coverage.

Response toxicity has no eval on this project

Overview

Plan

Review fixes for unreliable evals.

Hallucination eval gives no signal

Overview

Plan

Reduce unnecessary eval costs.

Exact-match check is running as an LLM judge

Overview

Plan

Automate recurring AI engineering work.

Find and investigate agent failures.

Find ways to reduce AI costs.

Start from a template, or build your own.

Give your evals a health check.

Subscribe to The Evaluator

Your evals need
evaluating, too.