Building Visibility Into SupportBot
You’re building a customer support agent called SupportBot. It classifies incoming queries, routes them to the right handler, and responds appropriately. But when something goes wrong, you’re left wondering: Why did the agent choose that tool instead of this one? What context was actually passed to the LLM when it generated that response? Without observability, debugging LLM applications feels like flying blind. In this chapter, you’ll learn how to instrument SupportBot with Arize AX to capture complete execution traces that answer these questions.What We’re Building
SupportBot has three core functions:- Classify incoming queries (order status vs. FAQ)
- Route to the appropriate handler
- Execute either database lookups or RAG-based knowledge searches
Follow with Complete Python Notebook
Prerequisites
Before starting, make sure you have:- An Arize AX account (sign up for free at app.arize.com)
- Your Space ID and API Key (found in Settings → API Keys)
- OpenAI API key (or another supported LLM provider)
Step 1: Install Dependencies
First, install the necessary packages for tracing:Step 2: Configure Tracing
Next, set up the connection to Arize AX. This is where all your traces will be sent.Step 3: Trace Your First LLM Call
Let’s start simple. Here’s how to classify a user query:What Just Happened?
- Created a parent span named “classify-query” that groups related operations
- Automatically captured the OpenAI call details (the instrumentation did this)
- Set OpenInference attributes to help Arize understand the span type and data
- Sent everything to Arize AX for visualization
Step 4: Add Tool Call Tracing
Now let’s add the ability to look up order status. We’ll use OpenAI’s function calling:Trace showing tool invocation nested under parent span
- The parent “handle-order-query” span
- The LLM call that decided to use a tool
- The “execute-tool” span with parameters and results
- The final LLM call that formulated the response
Step 5: Add RAG Tracing
For FAQ queries, we’ll use a simple RAG pipeline. Let’s trace both the retrieval and generation steps:RAG trace showing retrieval and generation steps
- What query was sent to retrieval
- Which documents were retrieved
- What context was passed to the LLM
- What answer was generated
Putting It All Together
Here’s the complete SupportBot with full tracing:What’s Next?
Tracing gives you visibility, but how do you measure quality at scale? In the next chapter, Annotations and Evaluations, you’ll learn how to:- Annotate traces directly in the Arize AX UI
- Run automated quality evaluations
- Identify systemic issues across thousands of traces
- Transform debugging from manual inspection into data-driven analysis