How to Write Your First AI Eval

In this workshop, SallyAnn DeLucia introduces practical ways to evaluate AI applications beyond informal vibe checks. She explains how evaluations differ from unit tests, surveys common evaluation approaches, and walks through reading a trace, defining pass or fail criteria, and building and iterating on an LLM judge in Arize’s prompt playground.

​

What you’ll learn

• Understand how evaluations complement unit tests for AI applications.
• Choose between LLM judges, decision models, code evaluations, annotations, and agent judges.
• Read an agent trace and turn expectations into clear pass or fail criteria.
• Build a dataset from traces and test an LLM judge.

Watch the workshop on YouTube.