The Evaluator
Your go-to blog for insights on AI observability and evaluation.
Showing 71–80 of 458 posts (page 8 of 46)
Beyond models: How context and evals make agents work in production
Building an AI agent has never been easier. But getting one into production that’s reliable is still hard. Most teams can ship a working demo in a day. The agent can answer questions, call tools, and even complete multi-step tasks. Then it hits production data, real users, and real constraints. That is where things break….
How to add an evaluation harness to your Gemini CLI coding agent
Coding agents can update prompts, wire in tools, and change application logic across your codebase in a single run. The hard part isn’t getting the agent to make changes, but knowing whether those changes actually made your LLM application better. Most teams still verify agent-driven changes with a few spot checks and intuition. That breaks…
Code is free, technical debt isn’t: Notes from AI Engineer Europe
Keynotes at Europe’s first flagship AI Engineer Conference shared one theme: code generation has accelerated past our ability to verify it, and the industry is quietly reorganizing around that fact. The smartest practitioners on stage and YC’s president are converging on the same answer. This post is a field report from three days in London…
Sign up for our newsletter, The Evaluator — and stay in the know with updates and new resources:
Data Fabric: Querying agent traces in BigQuery
How to join LLM traces with billing, infrastructure, and customer data using Iceberg and BigQuery If you run AI agents in production, you’ve probably run into a simple problem: you can’t answer basic questions about them in SQL. Your traces and evaluations live in your observability platform. Billing data lives in BigQuery. Infrastructure metrics live…
Building smarter AI agents: architecture, evals, and lessons from the field
Shipping an AI agent is easy. Understanding whether it actually works in production is not. That was the common thread across two AI Builders events in San Francisco at GitHub HQ and Seattle at The Collective, where a few hundred developers gathered to share what they’re learning from deploying agents. The AI Builders series is…
How Arize Skills Improved RAG Recall from 39% to 75% in 8 Hours
The Pain of Iterative RAG Development If you’ve built a production RAG system, you know this cycle. Tweak parameters, re-index, re-evaluate, repeat. It’s slow. It’s manual. The feedback loop between “I changed something” and “did it actually improve?” takes hours. This post is about what happened when I combined the Ralph autonomous agent pattern with…
From First Eval to Autonomous AI Ops: A Maturity Model for AI Evaluation
Every team runs evals. Almost none have an evaluation practice. The difference is the gap between a one-off notebook and a system that continuously assesses, alerts, and acts on what it finds. The teams that close that gap don’t do it in one leap — they grow into it. Here’s how we think about that…
100 AI Agents Per Employee: The Enterprise Governance Gap
100 AI agents per employee: The enterprise governance gap NVIDIA CEO Jensen Huang recently described a future where companies operate with roughly 100 AI agents per employee. That future is already starting to take shape. McKinsey reports 25,000 of its “employees” are agents, working alongside 60,000 humans. Then NVIDIA GTC happened. The company expanded its…
Managing Memory in AI Agents: Beyond the Context Window
This is part two of our deep dive series on how we built Alyx 2.0, our AI engineering agent. Watch us discuss it above, or keep reading for the full breakdown. Part one covered how we built planning into Alyx: the structured tools, the PlanMessage, and the enforcement gates that turned it from a task…
Why Banks Adopt the Arize Ecosystem
This post covers the organizational and regulatory patterns that shape AI platform decisions in banking, and why the Arize ecosystem aligns with how these institutions actually operate. Federated Architectures Large banks rarely operate as a single, centralized technology organization. Instead, they are structured as federated systems of business lines, each with its own priorities,…
