-
Agent EvaluationIs the cheap model good enough? How to test a new model in your agent
A paper called GVS5H claims a cheap model in a manager/worker loop closes the gap to frontier models. I tested that against Opus 5.5 in my own agent,… Jim Bennett November 5, 2026 21 min read -
Agent EvaluationI put a ghost in Claude Code. Then I evaluated it.
I built a Halloween Claude Code mod that grades each turn as a treat or a trick and blocks dangerous commands, then traced and evaluated it with Arize… Jim Bennett October 27, 2026 17 min read -
Agent EvaluationA web page talked our agent into leaking a stranger’s data. Here’s how we caught it.
A web page told our support agent to look up a different customer, and the agent complied on 28 of 100 runs. Here's how we found the leak… Nancy Chauhan October 26, 2026 15 min read -
Agent EvaluationHow to build a software factory that tests its own prompts
A small software factory that turns GitHub issues into released code, traces every station to Arize AX, and uses the verdicts it already produces to test and improve… Jim Bennett October 22, 2026 17 min read -
Agent EvaluationCan decision models play games from screenshots?
Decision models return a probability for each option in a fixed list. We tested whether image-capable decision models can play Tic-tac-toe, Minesweeper, and Flappy Bird from screenshots, and… Jim Bennett October 20, 2026 30 min read -
Agent EvaluationAgent tool selection: The tool your agent almost called
Decision spans in Arize Phoenix record a tool router’s choice, alternative probabilities, and confidence. Here’s how we evaluated those decisions and used the results to set a fallback… Nancy Chauhan October 19, 2026 14 min read -
Agent EvaluationAI agent permissions: Why agents don’t know what you’re allowed to do
AI agent permissions determine what an agent is allowed to do on someone’s behalf. As agents get more powerful, we’re running into a basic problem: a model can’t… Laurie Voss October 15, 2026 8 min read -
Agent EvaluationHow to test agent tool descriptions with Harbor and Phoenix
Build a reproducible benchmark, compare tool use across agents, and test whether a description change reduces unnecessary work while preserving outcomes. Nancy Chauhan October 12, 2026 11 min read -
Agent EvaluationAlignment is a human-AI workflow: how Custom Views make agent review possible
Automated evals can score every agent session, but they do not automatically reflect what your team considers a good outcome. That takes human judgment. Dheeraj Bandaru October 9, 2026 7 min read -
Agent EvaluationDecision model benchmark: Jev, Kev, Liquid d1, and more
On 15 September, TypeSafe released Jev, and within a few weeks about 10 companies had shipped something that works the same way. OpenAI, Liquid, Cloudflare, AWS and PostHog… Jim Bennett October 7, 2026 13 min read -
Agent EvaluationWhat agent traces can tell you without an LLM judge
Some agent failures are already in the trace. tracelint proves them on Phoenix OpenInference spans and fails CI, without an LLM judge. Ashwin Govind Ugale October 6, 2026 8 min read -
Agent EvaluationHow we benchmark AI agents and tools with Harbor and Arize Phoenix
See how Arize uses Harbor and Phoenix to benchmark AI agents, MCP servers, skills, and CLIs in reproducible sandboxes, then inspect traces and annotations. Elizabeth Hutton October 5, 2026 5 min read -
Agent EvaluationClaude’s hillclimb loop for AI agents: start with production traces
Anthropic's Claude hillclimb loop designs evals and improves agents one change at a time. Here's how to run that loop against production traces in Arize AX. Jim Bennett October 2, 2026 13 min read -
Agent EvaluationHow we built long-term memory for Alyx: why we chose a file over a knowledge graph
How we built long-term memory for Alyx: why we chose one 8,000-character file over retrieval and knowledge graphs, and how we tested it. Priyan Jindal Dheeraj Bandaru October 1, 2026 27 min read -
Agent EvaluationNVIDIA proposes an AI agent kill switch in silicon after a year of sandbox escapes
This year agents at OpenAI, Anthropic, and Google escaped environments meant to contain them. What decided severity was time-to-detect and time-to-kill, from 12 minutes to seven months. NVIDIA's… Jim Bennett September 30, 2026 13 min read
Don’t ship vibes.
Arize gives AI teams observability and evals to understand and improve agent performance.