The Evaluator
Your go-to blog for insights on AI observability and evaluation.
Showing 191–200 of 458 posts (page 20 of 46)
How to Build an AI Agent Router: Best Practices
Best practices for building an AI agent router including routing strategies, model selection, and evaluation patterns to send each request to the right agent. An AI agent router serves as the decision-making layer that manages how user requests are routed to the correct function, service, or action within a system. This component is particularly important…
Quick Guide to the EU AI Act for AI Teams
The EU AI Act is the world’s first comprehensive AI regulation, meant to promote responsible AI development and deployment in the European Union (EU). If you’re working with AI and have any interaction with the EU market, the regulations likely apply to you. We took a look at the Act to help unpack the risk…
Training Large Language Models to Reason in Continuous Latent Space
LLMs have traditionally been restricted to reason in the “language space,” where chain-of-thought (CoT) is used to solve complex reasoning problems. But a new paper argues that language space may not always be the best for reasoning. In this week’s paper read, we covered an exciting new technique from a team at Meta called Chain…
Sign up for our newsletter, The Evaluator — and stay in the know with updates and new resources:
Building Audio Support with OpenAI: Insights from our Journey
Introduction In early October last year, OpenAI launched the beta version of their Realtime API, which introduced an incredible feature: the ability to process audio as both input and output in a low latency environment. And when I say process audio, I don’t just mean converting it to text and passing it to an LLM—the…
Arize Release Notes: Voice Application Tracing and Evaluation
What’s New Voice Application Tracing and Evaluation Capture, process, and send audio data to Arize. Instrument your audio application to send events and traces to Arize, capture key events from the OpenAI Realtime API’s WebSocket and convert them into spans that provide meaningful insights into your system’s behavior. Assess how well your models identify emotional…
How Geotab and Arize AI Revolutionized Fleet Management with Generative AI
Geotab, a leader in fleet telematics, has taken a bold step forward in simplifying complex fleet data management. By leveraging generative AI, Geotab introduced its cutting-edge agent, Ace, designed to help fleet managers extract actionable insights from vast and complex datasets with ease. Here’s a closer look at the challenges they faced, the innovative solutions…
Arize Phoenix: 2024 in Review
2024 was Arize Phoenix‘s biggest year ever. Granted, it was also Phoenix’s first full year ever, but given how much we crammed into this year we think it still counts 🤗 Over the past year, Phoenix’s open-source LLM evaluation and tracing solution has grown from ~20k monthly downloads to over 2.5 million. Our community grew…
LLMs as Judges: A Comprehensive Survey on LLM-Based Evaluation Methods
We discuss a major survey of the LLMs-as-Judges paradigm: “LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods.” This paper systematically examines the LLMs-as-Judge framework across five dimensions: functionality, methodology, applications, meta-evaluation, and limitations. This gives us a birds eye view of the advantages, limitations and methods for evaluating its effectiveness. Read on for a summary…
Arize Release Notes: Prompt Hub, Managed Code Evaluators and More
Welcome to our regular update on new releases, enhancements, and changes. What’s New Prompt Hub The Prompt Hub is a centralized repository for managing, iterating, and deploying prompt templates within the Arize platform. It serves as a collaborative workspace for users to refine and store templates for various use cases, including production applications and experimentation….
How to Add LLM Evaluations to CI/CD Pipelines
In this post, we’ll explore how Continuous Integration and Continuous Deployment (CI/CD) can be used to evaluate large language models (LLMs) effectively. By integrating LLM evaluations into your CI/CD pipelines, you can ensure consistent, reliable AI performance and automate experimental results from your AI applications. Watch How to Set Up a CI/CD Pipeline for LLM…
