How We Scaled Support in Arize Copilot Without Slowing Down
Sally-Ann DeLucia
AI Product Manager
Arize Copilot has always had a clear vision: to empower AI engineers and data scientists to spend less time on repetitive tasks and more time building innovative applications. Copilot streamlines workflows, automates debugging, and surfaces actionable insights—all designed to help users move faster and achieve more.
With Copilot, we’ve always prioritized putting its capabilities where users need them most. From our experience, it’s clear that great AI applications can expand beyond chat, so while the chat window was our starting point, it quickly became clear that not all interactions fit into this format. With each new skill, we’ve moved further away from relying exclusively on chat and focused on embedding Copilot directly into the workflows it supports.
In this blog, we’ll share how strategic decisions allowed us to scale Copilot’s capabilities efficiently, including how partnering with an AI-powered support solution helped us quickly enhance technical support without pulling engineers away from core development.
Innovating with Our Expertise
With just two people dedicated to the project, we had to prioritize skills that were the best aligned with our expertise and that delivered the most value with the least lift.
Take our filter tool suite as an example. This skill integrates seamlessly into the filter bar, allowing users to interact with Copilot using natural language while staying within the context of their task. It doesn’t pull them out of their flow but instead meets them exactly where they need it. On the other hand, some skills—like our support skill—make perfect sense in the main chat window. Users might need help anywhere in the platform, so the chat is always available across every page. Here, users can ask questions, follow up as needed, and easily pick up where they left off.
Ultimately, our goal is to create purposeful integrations that enhance the user experience. Whether it’s a tool embedded in a specific workflow or a universally accessible chat, every Copilot skill is designed to fit naturally into how our users work.
Making Strategic Decisions to Stay Focused
While some features were achievable through prompt engineering and internal APIs, others—like technical support—required more than we could manage on our own. Early on, we knew that technical support would be a critical skill for Copilot. Our platform is highly technical, and new users often need guidance to make the most of it. Without fast, accurate answers, they could lose momentum—or worse, churn.
However, building an AI-powered support system from scratch would have required weeks of development time, pulling our engineers away from Copilot’s core capabilities. This led us to partner with RunLLM, an AI-powered technical support solution designed to handle complex user queries with precision.
From the very first test, RunLLM impressed us. We tested their assistant by having it write a GQL query for one of our monitors—a real use case—and it delivered the correct answer faster than expected. Within an hour of exploring their documentation, we had a fully functioning technical support skill integrated into Copilot. What would have taken over a month to build in-house was live in a matter of hours.
Delivering Impact with Copilot
Copilot adoption has been steadily growing, with sustained engagement and clear usage trends over the past several months. We’ve seen consistent increases in requests, with notable spikes in November and January, signaling that users are actively finding value in Copilot’s AI-powered assistance. While there are natural fluctuations, our most-used skills—Arize Support and Filter with AI—continue to drive engagement and deliver impact.
At the same time, we recognize there’s still room to grow. In the last six months, we’ve had around 700 unique users, and we’re focused on accelerating adoption by improving the UX, making Copilot more accessible in key workflows, and expanding its capabilities based on customer feedback. With new skills, deeper integrations, and a sharper focus on usability, we’re ensuring Copilot meets users where they need it most—helping them get to insights faster and more efficiently.
Looking Ahead
The AI landscape is changing faster than any of us can keep up with, but that’s part of our job at Arize. We’re constantly asking ourselves how to make AI engineers more productive and how we can automate away more tedious tasks. Part of that is knowing where to keep our focus – by choosing to partner strategically with a product like RunLLM, we delivered a high-quality product to our customers faster than we would have otherwise. That’s unlocked our time to work on key new features, and our roadmap is exciting: automatic debugging, deep insights, and tracing are all in the works. Stay tuned!