Instrument & Evaluate a Customer Support Agent

We’ll go through the following steps:

High Level Concepts

Trace your agent

Evaluate your agent

Next steps

High Level Concepts

In this example, we are building a customer support agent, which takes an input question from customers and decides what to do. In this example, we are deciding between searching for order information or answering the question.

Let’s break this down further: We are taking the user input and passing it to a router / planner prompt template. This template then decides which function call or agent skill to use. Often after calling a function, it goes back to the router template to decide on the next step, which could involve calling another agent skill.

Trace your agent

We have auto-instrumentation for function calling and structured outputs across almost every LLM provider here. We also support tracing for common frameworks using auto-instrumentation such as LangGraph, LlamaIndex Workflows, CrewAI, and AutoGen. To trace a simple agent with function calling, you can use arize-otel, our convenience package for setting up OTEL tracing along with openinference auto-instrumentation, which maps LLM metadata to a standardized set of trace and span attributes. Install the packages this guide uses:

pip install "arize<8" "arize-phoenix-evals>=3" arize-otel \
  openinference-instrumentation-openai openai pandas

Then set your credentials. Every block below reads them from the environment:

export ARIZE_SPACE_ID="<your-space-id>"
export ARIZE_API_KEY="<your-api-key>"
export OPENAI_API_KEY="<your-openai-api-key>"

Here’s some sample code on logging all OpenAI calls to Arize AX.

# Import open-telemetry dependencies
import os

from arize.otel import register

# Setup OTEL via our convenience function
tracer_provider = register(
    space_id=os.environ["ARIZE_SPACE_ID"],
    api_key=os.environ["ARIZE_API_KEY"],
    project_name="agents-cookbook",
)
# Import the automatic instrumentor from OpenInference
from openinference.instrumentation.openai import OpenAIInstrumentor

# Finish automatic instrumentation
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

Let’s create the foundation for our customer support agent. We have 2 functions that we define below: product_search, and track_package .

tools = [
    {
        "type": "function",
        "function": {
            "name": "product_search",
            "description": "Search for products based on criteria.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "The search query string.",
                    },
                    "category": {
                        "type": "string",
                        "description": "The category to filter the search.",
                    },
                    "min_price": {
                        "type": "number",
                        "description": "The minimum price of the products to search.",
                        "default": 0,
                    },
                    "max_price": {
                        "type": "number",
                        "description": "The maximum price of the products to search.",
                    },
                    "page": {
                        "type": "integer",
                        "description": "The page number for pagination.",
                        "default": 1,
                    },
                    "page_size": {
                        "type": "integer",
                        "description": "The number of results per page.",
                        "default": 20,
                    },
                },
                "required": ["query"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "track_package",
            "description": "Track the status of a package based on the tracking number.",
            "parameters": {
                "type": "object",
                "properties": {
                    "tracking_number": {
                        "type": "integer",
                        "description": "The tracking number of the package.",
                    }
                },
                "required": ["tracking_number"],
            }
        }
    }
]

We define a function below called run_prompt, which uses the chat completion call from OpenAI with functions, and returns the tool calls. Notice that we label tool_choice as required, so a function will always be returned.

import os
import openai
client = openai.Client()


def run_prompt(input):
    response = client.chat.completions.create(
        model="gpt-4.1-mini",
        temperature=0,
        tools=tools,
        tool_choice="required",
        messages=[
            {
                "role": "system",
                "content": " ",
            },
            {
                "role": "user",
                "content": input,
            },
        ],
    )

    if (
        hasattr(response.choices[0].message, "tool_calls")
        and response.choices[0].message.tool_calls is not None
        and len(response.choices[0].message.tool_calls) > 0
    ):
        return response.choices[0].message.tool_calls
    else:
        return []

Let’s test it and see if it returns the right function! If we ask a question about specific products, we’ll get a response that will call the product search function.

run_prompt("I'm interested in energy-efficient appliances, but I'm not sure which ones are best for a small home office. Can you help?")

ChatCompletionMessageToolCall(
    id='call_uAdifautsiR8VHnLRoGivjuk',
    function=Function(
        name='product_search',
        type='function',
        arguments='{"query":"energy-efficient appliances",
                    "category":"home office",
                    "page":1,
                    "page_size":5
                    }',
)

This results in a trace that looks like the following:

Evaluate your agent

Once we have generated a set of test cases, we can create evaluators to measure performance. This way, we don’t have to manually inspect every single trace to see if the LLM is doing the right thing. First we need a set of test questions to evaluate against. Generate them with one LLM call, then run each through the agent so every row has both the question and the agent’s response:

import pandas as pd

GEN_TEMPLATE = """
You are an assistant that generates complex customer service questions.
The questions should often involve:

- Multiple categories, so a question could logically fall into more than one
- Vague details that require clarification to categorize correctly
- Mixed intentions, where the customer's goal seems to conflict within the question
- Indirect phrasing that obscures the direct request, such as "I was wondering if..."

For Track Package, use vague timing references such as "recently" rather than dates.
For Product Search, use generic descriptors such as "energy-efficient appliances"
rather than specific product names.

Respond with one question per line. Do not number them. Generate 10 questions.
"""

resp = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "user", "content": GEN_TEMPLATE}],
)
questions = [q.strip() for q in resp.choices[0].message.content.strip().split("\n") if q.strip()]
questions_df = pd.DataFrame(questions, columns=["question"])

response_df = questions_df.copy(deep=True)
response_df["response"] = response_df["question"].apply(run_prompt).astype(str)

Now define the evaluation template that judges whether the router selected a function correctly. The same pattern extends to further checks, such as whether the right function was chosen and whether its arguments were filled correctly — see trace-level evaluation for a worked example running several judges together.

ROUTER_EVAL_TEMPLATE = """You are comparing a response to a question, and verifying whether that response should have made a function call instead of responding directly. Here is the data:
    [BEGIN DATA]
    ************
    [Question]: {question}
    ************
    [LLM Response]: {response}
    ************
    [END DATA]

Compare the Question above to the response. You must determine whether the reponse
decided to call the correct function.

Your response must be single word, either "correct" or "incorrect", and should not contain any text or characters aside from that word. "incorrect" means that the agent should have made function call instead of responding directly and did not, or the function call chosen was the incorrect one."correct" means the selected function would correctly and fully answer the user's question.

Here is more information on each function:
- product_search: Search for products based on criteria.
- track_package: Track the status of a package based on the tracking number."""

Run the judge with arize-phoenix-evals. create_classifier binds the template to a judge model and the labels it may return, and evaluate_dataframe scores every row of response_df:

from phoenix.evals import LLM, create_classifier, evaluate_dataframe

judge = LLM(provider="openai", model="gpt-4.1")

router_evaluator = create_classifier(
    name="router",
    llm=judge,
    prompt_template=ROUTER_EVAL_TEMPLATE,
    choices={"correct": 1.0, "incorrect": 0.0},
)

router_eval_df = evaluate_dataframe(
    dataframe=response_df,
    evaluators=[router_evaluator],
)

The template’s {question} and {response} placeholders are filled from the same-named columns in response_df, which is why the dataframe above uses exactly those names. evaluate_dataframe returns the original rows plus a router_score column holding a Score (its label, numeric score, and explanation). Below is a formatted example merged with the question and response.

Question	Response	Label	Explanation
Could you tell me about your selection of eco-friendly gadgets that might be suitable for a modern kitchen setup?	`ChatCompletionMessageToolCallname = 'product_search'type = 'function'arguments = { 'query': 'eco-friendly gadgets', 'category': 'kitchen'}`	correct	The user’s question is about finding eco-friendly gadgets suitable for a modern kitchen setup. The function call made is ‘product_search’ with the query ‘eco-friendly gadgets’ and category ‘kitchen’, which is appropriate for searching products based on the user’s criteria. Therefore, the function call is correct.

Next steps

We covered very simple examples of tracing and evaluating an agent that uses function calling to route user requests and take actions in your application. As you build more capabilities into your agent, you’ll need more advanced tooling to measure and improve performance. You can do all of the following in Arize AX:

Manually create tool spans which log your function calling inputs, latency, and outputs.
Evaluate your agent across multiple levels, not just the router prompt (more info on evaluations).
Create experiments to track changes across models, prompts, and parameters (more info on experiments).

Instrument

Observe

Evaluate

Improve

Advanced Workflows

Instrument & Evaluate a Customer Support Agent

High Level Concepts

Trace your agent

Evaluate your agent

Next steps

High Level Concepts

Trace your agent

Evaluate your agent

Next steps

High Level Concepts

Trace your agent

Evaluate your agent

Next steps

​High Level Concepts

​Trace your agent

​Evaluate your agent

​Next steps

High Level Concepts

Trace your agent

Evaluate your agent

Next steps