The Evaluator

AI Observability Integrations

Streamline and Centralize AI Analytics With Snowflake and Arize AI

This blog is co-authored by Aman Khan, Group Product Manager at Arize AI We’re thrilled to announce that Snowflake and Arize have joined forces to supercharge the machine learning (ML) toolchain and streamline how our joint customers access, analyze, and act on their machine learning model insights. The Snowflake Data Cloud enables customers to unite…

AI Evaluation AI Observability LLM Evaluation

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 Paper Reading

Introduction Recent research focuses on improving smaller models through imitation learning using outputs from large foundation models (LFMs). Challenges include limited imitation signals, homogeneous training data, and a lack of rigorous evaluation, leading to overestimation of small model capabilities. To address this, we introduce Orca, a 13-billion parameter model that learns to imitate LFMs’ reasoning…

AI Engineering

Interview: Mark Scarr, Senior Director of Data Science at Atlassian

Mark Scarr is the Senior Director of Data Science at Atlassian, where he heads up the Core Machine Learning Team. We talked to him about what the team is working on, Atlassian’s use cases, cloud migration, and what’s in store for the future. Please briefly introduce yourself and outline your role at Atlassian I lead…

AI Engineering AI Evaluation Research

One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning

Introduction In this week’s paper reading, we discuss “One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning.” GLoRA is a universal, parameter-efficient fine-tuning approach for diverse tasks. It enhances LoRA with a generalized prompt module, optimizing pre-trained model weights and activations. Its scalable, layer-wise structure search enables efficient parameter adaptation. GLoRA excels in transfer learning, few-shot learning, and…

AI Evaluation LLM Evaluation Research

HyDE: Precise Zero-Shot Dense Retrieval without Relevance Labels

Introduction In this paper reading, we explore HyDE: Precise Zero-Shot Dense Retrieval without Relevance Labels. HyDE is a thrilling zero-shot learning technique that combines GPT-3’s language understanding with contrastive text encoders. HyDE revolutionizes information retrieval and grounding in real-world data by generating hypothetical documents from queries and retrieving similar real-world documents. It outperforms traditional unsupervised…

LLM Evaluation

How To Troubleshoot LLM Summarization Tasks

This blog is co-authored by Xander Song, Developer Advocate at Arize Follow along in the Colab version of this blog Introduction Large language models (LLMs) are revolutionizing the field of AI, with over half of machine learning teams planning production deployments of LLMs over the next year. One of the most common use cases of…

Agent Engineering AI Engineering Research

Voyager: An Open-Ended Embodied Agent with LLMs Paper Reading and Discussion

Introduction In this paper reading, we discuss Voyager: the first LLM-powered embodied lifelong learning agent in Minecraft. Voyager autonomously explores the world, acquires skills, and makes discoveries without human intervention. It outperforms previous approaches, achieving exceptional proficiency in Minecraft, and successfully applies its learned skills to solve novel tasks in different Minecraft worlds, surpassing techniques…

AI Engineering AI Evaluation Research

LoRA: Low-Rank Adaptation of Large Language Models Paper Reading and Discussion

Introduction In this paper reading, we discuss LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Join us every Wednesday as we discuss the latest technical papers, covering a range of topics including large…

AI Evaluation LLM Evaluation Research

Retrieval-Augmented Generation – Paper Reading and Discussion

Introduction In this paper reading, we discuss “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” We know GPT-like LLMs are great at soaking up knowledge during pre-training and fine-tuning them can lead to some pretty great, specific results. But when it comes to tasks that really demand heavy knowledge lifting, they still fall short. Plus, it’s not…

Security & Governance

AI Ethical Issues Unraveled: Building a Fair, Transparent, and Responsible Future

In recent years, AI has permeated nearly every aspect of our lives, from healthcare and finance to education and entertainment. As AI technologies continue to advance, so do the ethical challenges associated with their development and implementation. What are AI Ethics? AI ethics refers to the moral principles guiding the design, development, and deployment of…