The Evaluator

AI Evaluation LLM Evals

AI Evals Maven Course Homework: the Recipe Bot Workflow

AI Evals for Engineers & PMs is a popular, hands‑on Maven course led by Hamel Husain and Shreya Shankar. The course’s goal is simple: “teach a systematic workflow for evaluating and improving LLM applications.” During the last cohort of the course, several guest lecturers were invited to speak and assign homeworks based on their open…

Agent Engineering AI Engineering

Claude Code vs. Cursor: A Power-User’s Playbook

Introduction If you spend your days hopping between Cursor’s VS-Code-style panels and Anthropic’s Claude Code CLI, you likely already intuitively know a key fact: while both promise AI-assisted development, they spring from opposite design philosophies. Cursor sprinkles AI into your editor; Claude Code lets AI drive your editor — and everything else on your machine….

AI Evaluation AI Observability LLM Observability

Annotation for Strong AI Evaluation Pipelines

This post walks through how human annotations fit into your evaluation pipeline in Phoenix, why they matter, and how you can combine them with evaluations to build a strong experimentation loop. Our notebook provides an easy way to test out these evaluations for yourself. Why Annotations Matter In AI  Annotations, specifically human annotations, provide high-quality…

AI Evaluation AI Product Quality Case Studies

How Handshake Deployed and Scaled 15+ LLM Use Cases In Under Six Months — With Evals From Day One

Handshake is the largest early-career network, specializing in connecting students and new grads with employers and career centers. It’s also an engineering powerhouse and innovator in applying AI to its product and features. Given constantly evolving model capabilities and user expectations, the product and engineering team at Handshake needed a way to ship LLM features…

LLM As A Judge LLM Evals LLM Evaluation

Evidence-Based Prompting Strategies for LLM-as-a-Judge: Explanations and Chain-of-Thought

When LLMs are used as evaluators, two design choices often determine the quality and usefulness of their judgments: whether to require explanations for decisions, and whether to use explicit chain-of-thought (CoT) prompting. Both affect not only evaluation accuracy but also the diagnostic value of the outputs.  The Role of Explanations in LLM Evaluation Asking for…

AI Observability LLM Evals LLM Evaluation

Trace-Level LLM Evaluations with Arize AX

Most commonly, we hear about evaluating LLM applications at the span level. This involves checking whether a tool call succeeded, whether an LLM hallucinated, or whether a response matched expectations. But sometimes, span-level metrics don’t tell the whole story. To understand if a workflow succeeded end-to-end, we need to zoom out to the trace level….

Agent Evaluation AI Evaluation LLM Evals

Session-Level Evaluations with Arize AX

When evaluating AI applications, we often look at things like tool calls, parameters, or individual model responses. While this span-level evaluation is useful, it doesn’t always capture the bigger picture of a user’s experience. That’s where session-level evaluations come in. In this post, we’ll walk through how to run session-level evaluations using the Arize Python…

AI Observability LLM As A Judge LLM Evals

LLM-as-a-Judge: Example of How To Build a Custom Evaluator Using a Benchmark Dataset

When To Build Custom Evaluators Arize-Phoenix ships with pre-built evaluators that are tested against benchmark datasets and tuned for repeatability. They’re a fast way to stand up rigorous evaluation for common scenarios. In practice, though, many teams work in specialized domains — such as medicine, finance, and agriculture — where models depend on proprietary data…

AI Observability Product Releases

adb Database: Realtime Ingestion At Scale

We put out our first blog on the introducing the Arize database – adb – in the beginning of July; this blog dives deeper into the realtime ingestion support of adb. AI data use-cases require an interface that supports both large files (like custom datasets) and real-time events from traces. The Arize AX platform was…

Agent Engineering Agent Observability AI Engineering

New In Arize AX: Prompt Learning, Arize Tracing Assistant, and Multiagent Visualization

July was a big month for Arize AX, with updates to make AI and agent engineering much easier. From prompt learning to new skills for Alyx and OpenInference Java, there is a lot to dive into. Here are some highlights on what we shipped. Alyx Updates At Arize:Observe, we debuted Alyx: an AI-powered assistant with…