Arize Blog & News | AI Observability and Evaluation Platform

Agent-as-a-Judge: Evaluate Agents with Agents

This week we dive into a paper that presents the “Agent-as-a-Judge” framework, a new paradigm for evaluating agent systems. Where typical evaluation methods focus solely on outcomes or demand extensive…

2 minutes read

By Sarah Welsh | 2 minutes read

Paper Readings Podcasts

Google’s NotebookLM and the Future of AI-Generated Audio

In this paper read, Aman Khan and Harrison Chu explore NotebookLM’s unique features, including its ability to generate realistic-sounding podcast episodes from text. Khan reviews NotebookLM from the product side,…

3 minutes read

By Sarah Welsh | 3 minutes read

Paper Readings Podcasts

Blog image Sally-Ann Delucia and Kyle O'Brien

Composable Interventions for Language Models

Introduction We’re excited to be joined by Kyle O’Brien, Applied Scientist at Microsoft, to discuss his most recent paper, Composable Interventions for Language Models. Kyle and his team present a…

33 minutes read

By Sarah Welsh | 33 minutes read

Paper Readings Podcasts Research

Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges

Introduction This week’s paper presents a comprehensive study of the performance of various LLMs acting as judges. The researchers leverage TriviaQA as a benchmark for assessing objective knowledge reasoning of…

39 minutes read

By Sarah Welsh | 39 minutes read

Paper Readings Podcasts Research

SallyAnn Delucia and Sai Kolsani headshots

RAFT: Adapting Language Model to Domain Specific RAG

Introduction Where adapting LLMs to specialized domains is essential (e.g., recent news, enterprise private documents), we discuss a paper that asks how we adapt pre-trained LLMs for RAG in specialized…

37 minutes read

By Sarah Welsh | 37 minutes read

Paper Readings Podcasts Research

Dat Ngo, Vibhu Sapra and Sai Kolsani headshots

LLM Interpretability and Sparse Autoencoders: Research from OpenAI and Anthropic

Introduction It’s been an exciting couple weeks for GenAI! Join us as we discuss the latest research from OpenAI and Anthropic. We’re excited to chat about this significant step forward…

42 minutes read

By Sarah Welsh | 42 minutes read

Paper Readings Podcasts Research

SallyAnn Delucia and Aparna Dhinakaran headshots

Breaking Down EvalGen: Who Validates the Validators?

Introduction Due to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating LLM outputs. Yet…

38 minutes read

By Sarah Welsh | 38 minutes read

Generative AI Large Language Models Paper Readings Podcasts Research

Keys To Understanding ReAct: Synergizing Reasoning and Acting in Language Models

Introduction This week we explore ReAct, an approach that enhances the reasoning and decision-making capabilities of LLMs by combining step-by-step reasoning with the ability to take actions and gather information…

38 minutes read

By Sarah Welsh | 38 minutes read

Paper Readings Podcasts Research

Demystifying Amazon’s Chronos: Learning the Language of Time Series

Introduction This week, we’ve covering Amazon’s time series model: Chronos. Developing accurate machine-learning-based forecasting models has traditionally required substantial dataset-specific tuning and model customization. Chronos however, is built on a…

35 minutes read

By Sarah Welsh | 35 minutes read

Paper Readings Podcasts Research Uncategorized

SallyAnne Delucia and Aman Khan headshots

Anthropic Claude 3

Introduction In this week’s Arize Community Paper Reading we dive into the latest buzz in the AI world—the arrival of Claude 3. Claude 3 is the newest family of models…

37 minutes read

By Sarah Welsh | 37 minutes read

Large Language Models Paper Readings Podcasts Research

Claire Longo and Duncan McKinnon headshots

Reinforcement Learning in the Era of LLMs

Introduction This week, we explore Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond, with Claire Longo, Arize’s…

37 minutes read

By Sarah Welsh | 37 minutes read

Large Language Models Paper Readings Podcasts Research Uncategorized

Sora: OpenAI’s Text-to-Video Generation Model

Introduction This week, we discuss the implications of Text-to-Video Generation and speculate as to the possibilities (and limitations) of this incredible technology with some hot takes. Dat Ngo, ML Solutions…

36 minutes read

By Sarah Welsh | 36 minutes read

Generative AI Large Language Models Paper Readings Podcasts Research

Sally-Ann DeLucia and Amber Roberts headshots

RAG vs Fine-Tuning

Introduction This week we discussed “RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture.” This paper that explores a pipeline for Fine-tuning and RAG, and presents the tradeoffs…

30 minutes read

By Sarah Welsh | 30 minutes read

Generative AI Large Language Models Paper Readings Podcasts Research

Blog

📚 Sign up for bi-weekly AI research paper readings

Resources featuring: “Podcasts”

Agent-as-a-Judge: Evaluate Agents with Agents

Google’s NotebookLM and the Future of AI-Generated Audio

Composable Interventions for Language Models

Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges

RAFT: Adapting Language Model to Domain Specific RAG

LLM Interpretability and Sparse Autoencoders: Research from OpenAI and Anthropic

Breaking Down EvalGen: Who Validates the Validators?

Keys To Understanding ReAct: Synergizing Reasoning and Acting in Language Models

Demystifying Amazon’s Chronos: Learning the Language of Time Series

Anthropic Claude 3

Reinforcement Learning in the Era of LLMs

Sora: OpenAI’s Text-to-Video Generation Model

RAG vs Fine-Tuning

Subscribe to our resources and blogs