The Evaluator
Your go-to blog for insights on AI observability and evaluation.
Showing 201–210 of 458 posts (page 21 of 46)
Merge, Ensemble, and Cooperate! A Survey on Collaborative LLM Strategies
LLMs have revolutionized natural language processing, showcasing remarkable versatility and capabilities. But individual LLMs often exhibit distinct strengths and weaknesses, influenced by differences in their training corpora. This diversity poses a challenge: how can we maximize the efficiency and utility of large language models? We discuss “Merge, Ensemble, and Cooperate: A Survey on Collaborative Strategies…
Arize Release Notes: Copilot Enhancements, Experiment Projects, and More
Welcome to our regular update on new releases, enhancements, and changes. What’s New Copilot Enhancements Span Chat The Copilot Span Chat skill makes getting value from spans faster and easier. Rather than spending time scrolling through and deciphering span data , teams can now: Analyze spans to extract key insights Ask questions to quickly understand…
AI Agent Workflows and Architectures Masterclass
While popular imagination and industry discourse can paint AI agents as complex autonomous systems with a mind of their own, practical implementations are far more straightforward. While specific definitions vary, an AI agent can be as simple as a program making one or more calls to a large language model. Production systems typically consist of…
Sign up for our newsletter, The Evaluator — and stay in the know with updates and new resources:
Building an AI Agent that Thrives in the Real World
Building an AI agent and keeping it running smoothly in production can feel like a daunting task. When it comes to working with LLMs, it’s still a bit of uncharted territory—most of us are figuring things out through trial and error. Even after you’ve launched in production, things can break or perform unexpectedly, forcing you…
Agent-as-a-Judge: Evaluate Agents with Agents
This week we dive into a paper that presents the “Agent-as-a-Judge” framework, a new paradigm for evaluating agent systems. Where typical evaluation methods focus solely on outcomes or demand extensive manual work, this approach uses agent systems to evaluate agent systems, offering intermediate feedback throughout the task-solving process. Agent-as-a-Judge enables scalable self-improvement. Among other things,…
Instrumenting Your LLM Application: Arize Phoenix and Vercel AI SDK
Instrumentation is an important tool for developers building with LLMs. It provides insight into application performance, behavior, and impact. This blog will cover: Why instrumentation matters for LLM applications Benefits of implementing instrumentation A guide on integrating Arize Phoenix with Vercel AI SDK for observability in Next.js applications Why Instrument Your LLM Application? 1. Performance…
What is AutoGen?
Thanks to Ali Saleh for his contributions to this piece. AutoGen is a framework that helps you easily create multi-agent applications. Multi-agent applications are a relatively recent idea that involve defining multiple LLM agents, each with their own goals and capabilities, and allowing them to work together to achieve an end goal. The multi-agent approach…
Introduction to OpenAI’s Realtime API
We break down OpenAI’s realtime API. Sally-Ann DeLucia and Aparna Dhinakaran cover how to seamlessly integrate powerful language models into your applications for instant, context-aware responses that drive user engagement. Whether you’re building chatbots, dynamic content tools, or enhancing real-time collaboration, we walk through the API’s capabilities, potential use cases, and best practices for implementation….
o1-preview Time Series Evaluations
Time series anomaly detection is one of the most challenging tasks we tackle at Arize. Using large language models (LLMs) for time series analysis, especially in our AI co-pilot assistant, has proven invaluable for uncovering intricate patterns, correlations, and potential issues in complex datasets. In this post, we’ll dive into the results of our recent…
Arize Release Notes: New Copilot Skills, Local Explainability, and More.
Welcome to our regular update on new releases, enhancements, and changes. What’s New New Copilot Skills Custom Metric Skill: Copilot now writes custom metrics! Users can generate their desired metric by having copilot translate natural language descriptions or existing code (e.g., SQL, Python) into AQL. Learn more Embedding Summarization Skill: Copilot now works for embeddings!…
