The Evaluator

Prompt Engineering Research

How to Prompt LLMs for Text-to-SQL

Introduction For this paper read, we’re joined by Shuaichen Chang, now an Applied Scientist at AWS AI Lab and author of this week’s paper to discuss his findings. Shuaichen’s research (conducted at the Ohio State University) investigates the impact of prompt constructions on the performance of large language models (LLMs) in the text-to-SQL task, particularly…

LLM Evals LLM Evaluation

Calling All Functions: Benchmarking OpenAI Function Calling and Explanations

This piece is co-authored by Roger Yang, Software Engineer at Arize AI Observability in third-party large language models (LLMs) is largely approached with benchmarking and evaluations since models like Anthropic’s Claude, OpenAI’s GPT models, and Google’s PaLM 2 are proprietary. In this blog post, we benchmark OpenAI’s GPT models with function calling and explanations against…

Integrations LLM Evals Prompt Engineering

Prompt Templates, Functions, and Prompt Window Management: Five Learnings From the Arize AI and PromptLayer Workshop

Introduction Prompt engineering is a crucial discipline that bridges the gap between raw model capabilities and practical, real-world applications. Recently, an enlightening event by Arize AI and PromptLayer titled “Prompt Templates, Functions, and Prompt Window Management” provided invaluable insights into this rapidly advancing domain. The speakers – Jared Zoneraich, Co-Founder of PromptLayer; Aparna Dhinakaran, Chief…

AI Engineering AI Evaluation Research

The Geometry of Truth: Emergent Linear Structure in LLM Representation of True/False Datasets

Introduction For this paper read, we’re joined by Samuel Marks, Postdoctoral Research Associate at Northeastern University, to discuss his paper, “The Geometry of Truth: Emergent Linear Structure in LLM Representation of True/False Datasets.” Samuel and his team curated high-quality datasets of true/false statements and used them to study in detail the structure of LLM representations…

AI Engineering AI Evaluation AI Observability

Ingesting Data for Semantic Searches in a Production-Ready Way

The current ecosystem around LLMs, semantic search and vector storage makes it easy to prototype but difficult to move into production. Ingesting large volumes of data specifically for embedding and upload to a vector database like Weaviate with high throughput and high reliability is difficult. Engineering teams should offload this responsibility to a vector embedding…

AI Engineering AI Evaluation Research

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Introduction In this paper read, we discuss “Towards Monosemanticity: Decomposing Language Models With Dictionary Learning,” a paper from Anthropic that addresses the challenge of understanding the inner workings of neural networks, drawing parallels with the complexity of human brain function. It explores the concept of “features,” (patterns of neuron activations) providing a more interpretable way…

AI Observability LLM Observability

Survey: Large Language Model Adoption Reaches Tipping Point

With a dizzying array of research papers and new tools, it’s an exciting time to be working at the cutting edge of AI. Given that the space is so new, some remain skeptical that LLM-powered applications are enterprise-ready – or that they can be deployed safely and reliably. How common is this concern – and…

AI Observability AI Product Quality

AI ROI: Guide To Observability Value Statistics

Introduction Due to its unique ability to preemptively detect and fix model issues that may be impacting business value, model observability initiatives often yield a high return on investment (ROI). At Arize, we think of a “model insight” as an issue with a model or data that has an impact on the model performance in…

Open Source Research

RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models

Introduction In this paper reading, we’ll be discussing RankVicuna, the first fully open-source LLM capable of performing high-quality listwise reranking in a zero-shot setting. While researchers have successfully applied LLMs such as ChatGPT to reranking in an information retrieval context, such work has mostly been built on proprietary models hidden behind opaque API endpoints. This…

Security & Governance

Implementing Text PII Anonymization

This piece is co-authored by Ilya Reznik (Medium; Contact) Introduction While technology makes it very easy to share information, it also makes it easy to share all sorts of information in unintended ways. Case in point: personally identifying information (PII) can surface in software systems in unexpected ways. Whether it is logging or storing data…