The Evaluator
Your go-to blog for insights on AI observability and evaluation.
3 production patterns for AI agents and how to evaluate each one
A local coding agent, an in-app customer assistant, and an AI SRE triaging production logs may all use the same model class—but not the same harness, eval plan, or rollout risk. Mastra CEO Sam Bhagwat breaks down the three production patterns and how to evaluate each one.
What is a loop in AI engineering, anyway?
The AI engineering world is using “loop” to describe several different agent architectures. This post maps execution loops, task loops, product loops, system loops, and the human oversight loop that controls them.
Trace before you migrate: Measuring Kubernetes bottlenecks in AI agent sandboxes
Kubernetes is strong for long-lived services, but it is often a poor default for short-lived agent sandboxes. Trace sandbox creation, tool execution, eval latency, and full trajectory time before you migrate runtimes.
Sign up for our newsletter, The Evaluator — and stay in the know with updates and new resources:
The agent is the user now: lessons from the founder of WorkOS
WorkOS founder Michael Grinich explains why the next era of AI engineering depends on the systems around agents: identity, permissions, evals, memory, and feedback loops that keep autonomous software from succeeding in the wrong ways.
Evals in CI: How to write your LLM evals as tests with Arize Phoenix
If you’re struggling to get started with evals, you’re not alone. This post explains how to write LLM evals as ordinary tests in CI with Phoenix, pytest, and Vitest/Jest.
Own the loop: A field guide to agent harnesses
As models become cheaper and more interchangeable, the durable advantage shifts to the agent harness: the loop, tools, memory, permissions, and workflow you can own and refine.
How to evaluate AI agents, avoid reward hacking, and build better specs
Agent evals are repeatable tests that score whether AI agents completed a task correctly. Learn how to design rubrics, test suites, and trace-based evals that catch failures and prevent reward hacking.
Model subsidies are ending. What do you do now?
Flat-rate AI plans are subsidizing agentic workloads. Learn why LLM inference costs are moving to metered pricing and how evals reveal cost per successful task.
AI evals are a data science problem: What most teams get wrong
Hamel Husain explains why the best AI teams treat LLM judges like classifiers, not dashboards.
Trace and evaluate TrueFoundry AI Gateway traffic in Arize AX
Learn how TrueFoundry AI Gateway exports OpenTelemetry traces to Arize AX so teams can trace, evaluate, and monitor production LLM and agent traffic without embedding a vendor SDK in every service.
