The Evaluator

AI Product Quality Case Studies

How Flipkart Leverages Generative AI for 600 Million Users

Catering customer support for 600 million users is a feat in itself. Between sessions at this year’s Arize:Observe, Flipkart’s Anusua (Anu) Trivedi talked to Aparna Dhinakaran about the company’s challenges and generative AI use cases. Among other things, Anu talked about how tricky measuring success can be. Trivedi explains that “you have to define it…

Agent Engineering Agent Evaluation AI Engineering

LlamaIndex Workflows: Navigating a New Way To Build Cyclical Agents

Last week, LlamaIndex released Workflows, a new approach to easily create agents. Workflows use an event-based architecture instead of the directed acyclic graph approach used by traditional pipelines or chains. This new approach brings with it new considerations for developers looking to create agentic systems, as well as new questions on how to evaluate and…

Agent Observability AI Observability LLM Observability

Arize Release Notes: Aug 8, 2024

Welcome to our regular update on new releases, enhancements, and changes. What’s New Auto Instrumentation Automatically collect traces from an expanded set of frameworks and libraries. Haystack LiteLLM CrewAI PromptFlow Groq Examples If helpful or illustrative, example notebooks are available from OpenInference on everything from RAG pipelines to building fallbacks with conditional routing with Haystack,…

AI Engineering AI Evaluation Research

Breaking Down Meta’s Llama 3 Herd of Models

Introduction Meta just released Llama 3.1 405B–and according to them, it’s “the first openly available model that rivals the top AI models when it comes to state-of-the-art capabilities in general knowledge, steerability, math, tool use, and multilingual translation.” Will the latest Llama herd ignite new applications and modeling paradigms like synthetic data generation? Will it…

LLM As A Judge LLM Evals LLM Evaluation

Text To SQL: Evaluating SQL Generation with LLM as a Judge

Special shoutout to Manas Singh for collaborating with us on this research! One application of LLMs that has garnered headlines and significant investment surrounds their ability to generate SQL queries. The ability to query large databases with natural language unlocks several compelling use cases on everything from greater data transparency to increasing accessibility for non-technical…

AI Evaluation AI Observability LLM Evaluation

Arize AI: Support for EU Data Residency

Arize AI recently rolled out EU data residency for all users, enabling customers to host their data within the European Union. By offering EU data residency, Arize enables organizations to use its AI observability and LLM evaluation tools while adhering to local data protection laws. Companies can comply with the EU’s General Data Protection Regulation…

Agents AI Engineering

Developing Copilot: What AI Engineers Can Learn from Our Experience Building An AI Assistant

Arize Copilot began as an ambitious idea: to develop an AI assistant tailored specifically for data scientists and AI engineers. This tool was designed to assist in streamlining their workflows and simplifying complex tasks. Throughout its development, we encountered numerous challenges and learned valuable lessons about developing with LLMs. Copilot is an interactive tool that…

AI Observability LLM Observability

Different Ways to Instrument Your LLM Application

Thanks to John Gilhuly for his contributions to this piece LLM instrumentation is the process of monitoring and collecting data in an LLM application, and it plays an important role in achieving the level of performance and reliability necessary in these systems. This blog explores the different ways you can instrument your LLM application, comparing…

Prompt Engineering Research

DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines

Introduction Chaining language model (LM) calls as composable modules is fueling a new way of programming, but ensuring LMs adhere to important constraints requires heuristic “prompt engineering.” The paper this week introduces LM Assertions, a programming construct for expressing computational constraints that LMs should satisfy. The researchers integrated their constructs into the recent DSPy programming…

LLM Evals LLM Evaluation

LLM Function Calling: Evaluating Tool Calls In LLM Pipelines

Function calling is an essential part of any AI engineer’s toolkit, enabling builders to enhance a model’s utility at specific tasks. As more LLM applications leveraging tool calls get deployed into production, the task of effectively evaluating their performance in LLM pipelines becomes more critical. What Is Function Calling In AI? First launched by OpenAI…