The Evaluator

AI Evaluation LLM Evals LLM Evaluation

Breaking Down EvalGen: Who Validates the Validators?

Introduction Due to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating LLM outputs. Yet LLM-generated evaluators often inherit the problems of the LLMs they evaluate, requiring further human validation. This week’s paper explores EvalGen, a mixed-initative approach to aligning…

Agent Engineering Research

Keys To Understanding ReAct: Synergizing Reasoning and Acting in Language Models

Introduction This week we explore ReAct, an approach that enhances the reasoning and decision-making capabilities of LLMs by combining step-by-step reasoning with the ability to take actions and gather information from external sources in a unified framework. There is also a ReAct demo with a colab, and a deep dive into reflexion. Watch Dive in…

Agent Engineering AI Engineering Open Source

Four Tips on How To Read AI Research Papers Effectively

According to a recent survey, over two-thirds (66.9%) of developers and machine learning teams are planning production deployments of LLM apps in the next 12 months or “as fast as possible” – and 14.1% are already in production! Given the rapid rate of progress and constant drumbeat of new foundation models, orchestration frameworks and open…

AI Engineering AI Evaluation Research

Demystifying Amazon’s Chronos: Learning the Language of Time Series

Introduction This week, we’ve covering Amazon’s time series model: Chronos. Developing accurate machine-learning-based forecasting models has traditionally required substantial dataset-specific tuning and model customization. Chronos however, is built on a language model architecture and trained with billions of tokenized time series observations, enabling it to provide accurate zero-shot forecasts matching or exceeding purpose-built models. We…

AI Engineering AI Evaluation Research

Anthropic Claude 3

Introduction In this week’s Arize Community Paper Reading we dive into the latest buzz in the AI world—the arrival of Claude 3. Claude 3 is the newest family of models in the LLM space, where Opus Claude 3—Anthropic’s “most intelligent” Claude model—challenges the likes of GPT-4. According to Anthropic, the Claude 3 family of models…

LLM Evals Open Source

How To Set Up a SQL Router Query Engine for Effective Text-To-SQL

This article co-authored by Dustin Ngo Large language model (LLM) applications are being deployed by an increasing number of companies to power everything from code generation to improved summarization of customer service calls. One area where LLMs with in-context learning show promise is text-to-SQL, or generating SQL queries from natural language. Achieving results is often…

AI Engineering AI Product Quality Prompt Engineering

Reinforcement Learning in the Era of LLMs

Introduction This week, we explore Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond, with Claire Longo, Arize’s Head of Customer Success and Duncan McKinnon, ML Solutions Engineer at Arize. Recent advancements in Large Language Models (LLMs) have garnered wide attention and led…

Agents AI Evaluation LLM Evals

Evaluate RAG with LLM Evals and Benchmarks

Recently, I attended a workshop organized by Arize AI titled “RAG Time! Evaluate RAG with LLM Evals and Benchmarking.” Hosted by Amber Roberts – ML Growth Lead at Arize AI, and Mikyo King – Head of Open Source at Arize AI, the talks provided valuable insights into an important field of study. Miss the event?…

AI Engineering AI Evaluation LLM Evals

Sora: OpenAI’s Text-to-Video Generation Model

Introduction This week, we talk about the implications of Text-to-Video Generation and speculate as to the possibilities (and limitations) of this incredible technology with some hot takes. Dat Ngo, ML Solutions Engineer at Arize, is joined by AI Engineer Vibhu Sapra to break down OpenAI’s technical report on their Text-To-Video Generation Model: Sora. According to…

Generative AI Large Language Models LLM Evals

Sora: OpenAI’s Text-to-Video Generation Model

Introduction This week, we discuss the implications of Text-to-Video Generation and speculate as to the possibilities (and limitations) of this incredible technology with some hot takes. Dat Ngo, ML Solutions Engineer at Arize, is joined by AI Engineer Vibhu Sapra to discuss OpenAI’s technical report on their Text-To-Video Generation Model: Sora. According to OpenAI, “Sora…