Skip to main content
Claude Managed Agents runs your agent loop on Anthropic’s servers and executes its tools in a sandbox that Anthropic hosts. Your application starts a session, sends it messages, and answers the custom tools that only your application can run. Most of the work happens where your own tracing can’t see it. You need traces from both sides to see what the agent did and why. The Claude Platform exports OpenTelemetry spans for the work it does, and the Claude SDK emits spans for the requests your application sends. In this guide you send both to Arize AX, join them into a single trace, and then evaluate the agent’s answers.
OpenTelemetry export from the Claude API is now in public beta. Span names and attributes can change during the beta.
This guide covers Anthropic’s Claude Managed Agents product. For the managed agents that run inside Arize AX, see Connect your harness.

Overview

You will build a small issue-triage agent. It reads a project’s open issues through a custom list_issues tool that your application runs, counts them by label with bash in its sandbox, and names the most urgent one. You trace a session from end to end and read the trace in Arize AX. Then you add an evaluator that checks whether the agent picked the right issue. This guide is for developers who build agents on Claude Managed Agents and want to observe and evaluate them in Arize AX. You should be comfortable with:
  • Python, TypeScript, Go, or Java
  • The Managed Agents concepts of agents, environments, sessions, and events. The Managed Agents quickstart covers them.
  • Reading traces in Arize AX
By the end, you will be able to:
  • Send the spans that the Claude Platform creates to an Arize AX project
  • Join those spans with your application’s spans into one trace
  • Trace the custom tools that your application runs for the agent
  • Evaluate the agent’s answers with an LLM-as-a-judge task

Before you start

You need:
  • Python 3.10+, Node.js 18.19+, Go 1.26+, or Java 17+
  • An Arize AX account (sign up), with your Space ID and API Key from Space Settings
  • The admin role in a Claude Console organization, to add a destination
  • An ANTHROPIC_API_KEY from the Claude Console

Send Claude Platform spans to Arize AX

The Claude API exports its spans to destinations that you add in the Claude Console. Add one that points at the Arize AX project for this guide:
1

Open the OpenTelemetry settings

In the Claude Console, open the OpenTelemetry settings page and click Add destination.
2

Choose Arize AX

Set Provider to Arize, set Product to Arize AX, and set Region to the region of your Arize AX space.
3

Enter your Arize AX details

Enter your Space ID and API key, and set Project name to claude-managed-agents-triage.
4

Set the content mode

Set Content mode to Full content. The default, Metadata only, leaves out the prompts, messages, and tool arguments that this guide reads and evaluates.
The Add destination form in the Claude Console with Provider set to Arize, Product set to Arize AX rather than Arize Phoenix, Name set to Arize AX, and Region set to US. The Space ID and API key are filled in and blurred, the project name is claude-managed-agents, and OTLP details are expanded to show the endpoint https://otlp.arize.com and the headers arize-space-id, arize-api-key, and x-project-name. Content mode is set to Full content, Workspaces is set to All rather than Selected or All except, and Advanced settings is collapsed.

The Claude Console's Add destination form with the Arize provider and the Arize AX product

5

Choose the workspace

Leave Workspaces at All to export from every workspace in your organization, or choose Selected and select the workspace that your API key belongs to. Then click Add destination.
6

Send a test span

In the destination’s More actions menu, click Test. In Arize AX, check that an anthropic.test span arrived in the claude-managed-agents-triage project.
The details panel of an Arize AX destination in the Claude Console, showing the endpoint https://otlp.arize.com, content mode Full content, delivery Healthy, and one workspace. The More actions menu is open with Edit, Test, Disable, and Delete, and Test is highlighted.

The destination's More actions menu in the Claude Console, with Test highlighted

A destination exports every request made with that workspace’s API keys, including Messages API requests from other tools, so use a workspace that runs only this guide’s agent if you want a clean project. The project name decides which project the spans land in, because the Claude API sets the resource attributes of its spans itself. For the details of each field, see the Claude Managed Agents integration.

Set up tracing in your application

Your application needs an OpenTelemetry tracer provider that exports to the same Arize AX project. The Claude SDK uses the tracer provider that your application registers, so you don’t pass it to the client.
1

Install the packages

Install the Claude SDK and the tracing packages for your language:
Install or upgrade the Claude SDK to a version that emits spans. The SDK emits a span for each request and sends the trace context that joins the API’s spans to yours.
2

Set your credentials

ARIZE_PROJECT_NAME must match the destination’s Project name, so the two sources of spans meet in one project:
ANTHROPIC_OPEN_TELEMETRY_TRACES_CONTENT_MODE=content makes the SDK’s spans carry prompts and responses, matching the Full content mode of the destination. The programs send to the US endpoint by default. For a space in another region, also set ARIZE_COLLECTOR_ENDPOINT to that region’s endpoint, for example https://otlp.eu-west-1a.arize.com/v1 for EU spaces.
3

Register the tracer provider

For Python and TypeScript, register the tracer provider in its own module, so that it runs before the Claude SDK makes its first request:
For Go and Java, the same setup is at the top of main in Build the triage agent. It also registers the W3C trace context propagator, which neither language registers by default. The SDK uses the propagator to send each span’s trace context to the Claude API, which then nests its own spans under yours.

Build the triage agent

The program has three functions:
  • create_agent creates an environment and an agent. The agent has Anthropic’s built-in toolset, with the web tools turned off, and one custom tool, list_issues. A custom tool runs in your application, not in the sandbox. That makes it the right place for anything that needs your data or your credentials.
  • list_issues answers the custom tool from a small list of sample issues. It wraps that work in its own TOOL span, with OpenInference’s @tracer.tool decorator in Python and its traceTool wrapper in TypeScript. In Python, @tracer.tool reads the tool’s name, description, and parameters from the function’s name, docstring, and signature. TypeScript, Go, and Java set them on the span themselves. The Claude API sees the call and the result but not your code, so this span is the only record of what your application did. If you run custom tools with the Claude SDK’s session tool runner instead, the SDK creates this TOOL span for you, so don’t add your own.
  • run_triage starts a session inside a triage-request CHAIN span, created with @tracer.chain in Python and traceChain in TypeScript, sends the request, and reads the session’s event stream. When the agent calls list_issues, the session pauses until your application sends back the result. The loop ends when the session goes idle for any reason other than requires_action, which means it’s waiting on you.
The triage-request span records the request as its input and the agent’s answer as its output. The Python and TypeScript helpers record them from the function’s argument and return value, and Go and Java set the input.value and output.value attributes themselves. The evaluator in Evaluate the triage scores those two values. Every span you create also sets its status: OK on success, or an error with the exception if a request fails.
Agents and environments are persistent and versioned. The program creates them on each run to stay self-contained. In your own application, create them once, store their IDs, and pass the IDs to each sessions.create call.

Run a session

Run the program:
The agent calls list_issues, counts the issues by label with bash in its sandbox, and replies. The wording and layout vary from run to run, but the output looks similar to this:

Explore the trace

1

Open the trace

Open your Arize AX space and select the claude-managed-agents-triage project. A new trace appears within about a minute. The API’s spans can arrive after yours. Open the trace to see the session from both sides:
  • triage-request is your root span, with the request as input and the agent’s answer as output.
  • Under it are the SDK’s CLIENT spans, one for each request your application sent. Each one contains the API’s span for the same request, with the same name.
  • Under the request that sent your message is the agent’s first turn. It ends with the stop reason requires_action when the agent calls list_issues. Your list_issues TOOL span follows, then the request that sent the result, and a second turn in which the agent runs bash and answers.
  • The SDK’s spans for creating the environment and the agent are in traces of their own, because the program creates them before it starts triage-request.
Arize AX trace view of a triage-request trace. The tree shows anthropic.sessions.create, anthropic.sessions.events.stream, and anthropic.sessions.events.send spans, a first anthropic.session.turn with one model request, the list_issues tool span, then a second turn with two model requests and an anthropic.tool_use bash span. The selected root span shows the triage request as input and the agent's answer, a table of open issues by label naming issue 101 as most urgent, as output.

The triage session's trace, with your spans, the SDK's spans, and the API's spans in one tree

2

Read the agent's turns

Open an anthropic.session.turn span to read the system prompt and the turn’s input and output messages, including the tool calls and their results. In the first turn, the agent’s output is the list_issues call:
The first anthropic.session.turn span selected in Arize AX. Its input messages are the system prompt telling the agent to call list_issues and use bash, and the user's triage request. Its output message is a list_issues call with state set to open.
3

Inspect a model request and a tool call

Open an anthropic.model_request span to see the model and token usage of one model request. Open the bash tool span to see the command that the agent ran in its sandbox. Its arguments are the span’s input:
The anthropic.tool_use bash span selected in Arize AX. Its input is a bash command that pipes the issue list into a Python one-liner to count issues by label, and the bash tool's parameters show the same command as JSON.
The command’s result is in the span’s anthropic.tool_result.content attribute, on the Attributes tab:
The Attributes tab of the anthropic.tool_use bash span in Arize AX. The anthropic.tool_result.content attribute holds the command's output, the counts 2 bug, 1 auth, 1 docs, 1 enhancement, and 1 export, alongside the session ID, the tool's evaluated permission of allow, and the workspace ID.
4

Read the session as a conversation

The API’s spans also carry the session’s thread ID in gen_ai.conversation.id, and Arize AX uses it as the session ID, so each thread appears in the project’s Sessions tab. Open Sessions and select the session to read the thread as a conversation.
An Arize AX session for a Claude Managed Agents thread, on the Session Conversation tab. Turn 1 shows the user's triage request and the assistant's reply with a table of open issues by label.
The Sessions list shows no first message, last message, or duration for these sessions. Arize AX fills in those columns from the root span of each trace in a session, and the root here is your triage-request span. The session ID is only on the SDK’s and the API’s spans further down the trace, so the list has nothing to summarize. Selecting the session still shows the whole conversation.A Managed Agents session can have more than one thread, for example when its agent hands work to other agents. Each thread is a separate session in Arize AX, but every span in every thread carries the same anthropic.session.id. To see all of a session’s work in one place, filter the project’s spans on anthropic.session.id. A turn that starts because an agent on another thread replied or finished has no parent, so it’s the root of a trace of its own.

Evaluate the triage

The trace shows what the agent did. An evaluator tells you whether it did it well. Sample issue #101 locks out a whole region, so a good triage names it as the most urgent. Create an LLM-as-a-judge evaluator that checks the agent’s answer against that:
1

Create the task

In Arize AX, open Evaluators and click New Task. Name the task triage-quality and select the claude-managed-agents-triage project.
2

Filter to your root spans

Add a filter on the span name triage-request, so the task scores your root spans.
3

Add the evaluator

Click Add Evaluator, choose Create Custom, and use this template:
Map input to attributes.input.value and output to attributes.output.value, and set the choices to correct (score 1) and incorrect (score 0).
4

Run the task

Turn on Run continuously, and run it once on your existing traces as well. Then click Create Task.
The task panel in Arize AX for a task named triage-quality. The template section shows the judge model and the triage prompt with input and output variables, and the labels correct with score 1 and incorrect with score 0. The configuration section shows the project data source, the span query name = 'triage-request', five matching CHAIN spans whose input is the triage request, and input mapped to attributes.input.value.

The triage-quality task: the judge template, its labels, the span filter, and the matching triage-request spans

5

Read the results

Run the program a few more times. Each triage-request span gets a triage-quality label.
The Traces tab of an Arize AX project filtered to name = 'triage-request', showing five traces whose Span Evaluations column reads triage-quality: correct.

Triage traces with their triage-quality labels

Filter the project on incorrect to find the sessions where the agent got it wrong, and open their traces to see whether the cause was the tool result, the bash step, or the model’s reasoning. For the other ways to create a task, such as with Arize Skills or Alyx, and for the full set of options, see Run evals on traces.

Summary

You traced a Claude Managed Agents session from both sides and evaluated its result:
  • A Claude Console destination sends the spans that the Claude API creates to an Arize AX project, routed by the destination’s project name.
  • The Claude SDK’s spans and W3C trace context join those spans to your application’s own trace.
  • Your own TOOL span records the custom tool that your application runs, which the Claude API can’t see.
  • An LLM-as-a-judge task scores each triage, so you can find the runs that went wrong and trace their cause.

Next steps

Claude Managed Agents integration

The reference for the Console destination, the SDK setup, and the spans Arize AX captures.

Session-level evaluations

Score whole conversations instead of single answers.

Evaluate agents

Score tool selection, tool results, and goal completion.

GenAI semantic conventions

How Arize AX converts gen_ai.* spans to OpenInference.