> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Trace and Evaluate Claude Managed Agents
> Build an issue-triage agent on Claude Managed Agents, trace each session in Arize AX with OpenTelemetry, and evaluate the agent's triage decisions.
[Claude Managed Agents](https://platform.claude.com/docs/en/managed-agents/overview) runs your agent loop on Anthropic's servers and executes its tools in a sandbox that Anthropic hosts. Your application starts a session, sends it messages, and answers the custom tools that only your application can run. Most of the work happens where your own tracing can't see it. You need traces from both sides to see what the agent did and why.
The Claude Platform exports OpenTelemetry spans for the work it does, and the Claude SDK emits spans for the requests your application sends. In this guide you send both to Arize AX, join them into a single trace, and then evaluate the agent's answers.
OpenTelemetry export from the Claude API is now in public beta. Span names and attributes can change during the beta.
This guide covers Anthropic's **Claude Managed Agents** product. For the managed agents that run inside Arize AX, see [Connect your harness](/docs/ax/agents/connect-your-harness).
## Overview
You will build a small issue-triage agent. It reads a project's open issues through a custom `list_issues` tool that your application runs, counts them by label with bash in its sandbox, and names the most urgent one. You trace a session from end to end and read the trace in Arize AX. Then you add an evaluator that checks whether the agent picked the right issue.
This guide is for developers who build agents on Claude Managed Agents and want to observe and evaluate them in Arize AX. You should be comfortable with:
* Python, TypeScript, Go, or Java
* The Managed Agents concepts of agents, environments, sessions, and events. The [Managed Agents quickstart](https://platform.claude.com/docs/en/managed-agents/quickstart) covers them.
* Reading traces in Arize AX
By the end, you will be able to:
* Send the spans that the Claude Platform creates to an Arize AX project
* Join those spans with your application's spans into one trace
* Trace the custom tools that your application runs for the agent
* Evaluate the agent's answers with an LLM-as-a-judge task
## Before you start
You need:
* Python 3.10+, Node.js 18.19+, Go 1.26+, or Java 17+
* An Arize AX account ([sign up](https://arize.com/sign-up/)), with your **Space ID** and **API Key** from **Space Settings**
* The **admin** role in a Claude Console organization, to add a destination
* An `ANTHROPIC_API_KEY` from the [Claude Console](https://platform.claude.com/)
## Send Claude Platform spans to Arize AX
The Claude API exports its spans to *destinations* that you add in the Claude Console. Add one that points at the Arize AX project for this guide:
In the Claude Console, open the [OpenTelemetry settings page](https://platform.claude.com/settings/telemetry-destinations) and click **Add destination**.
Set **Provider** to **Arize**, set **Product** to **Arize AX**, and set **Region** to the region of your Arize AX space.
Enter your **Space ID** and **API key**, and set **Project name** to `claude-managed-agents-triage`.
Set **Content mode** to **Full content**. The default, **Metadata only**, leaves out the prompts, messages, and tool arguments that this guide reads and evaluates.
Leave **Workspaces** at **All** to export from every workspace in your organization, or choose **Selected** and select the workspace that your API key belongs to. Then click **Add destination**.
In the destination's **More actions** menu, click **Test**. In Arize AX, check that an `anthropic.test` span arrived in the `claude-managed-agents-triage` project.
A destination exports every request made with that workspace's API keys, including Messages API requests from other tools, so use a workspace that runs only this guide's agent if you want a clean project. The project name decides which project the spans land in, because the Claude API sets the resource attributes of its spans itself. For the details of each field, see the [Claude Managed Agents integration](/docs/ax/integrations/platforms/claude-managed-agents/claude-managed-agents-tracing#export-spans-from-the-claude-console).
## Set up tracing in your application
Your application needs an OpenTelemetry tracer provider that exports to the same Arize AX project. The Claude SDK uses the tracer provider that your application registers, so you don't pass it to the client.
Install the Claude SDK and the tracing packages for your language:
```bash Python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
pip install arize-otel anthropic
```
```bash TypeScript theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
npm install @anthropic-ai/sdk \
@opentelemetry/api \
@opentelemetry/exporter-trace-otlp-proto \
@opentelemetry/resources \
@opentelemetry/sdk-trace-base \
@opentelemetry/sdk-trace-node \
@opentelemetry/semantic-conventions \
@arizeai/openinference-core \
@arizeai/openinference-semantic-conventions \
tsx
```
```bash Go theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
go get \
github.com/Arize-ai/arize-otel-go \
github.com/anthropics/anthropic-sdk-go
```
```groovy Java theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
// build.gradle
plugins {
id 'application'
}
repositories {
mavenCentral()
}
dependencies {
implementation 'com.anthropic:anthropic-java:2.72.0'
implementation 'io.opentelemetry:opentelemetry-sdk:1.66.0'
implementation 'io.opentelemetry:opentelemetry-exporter-otlp:1.66.0'
}
application {
mainClass = 'example.Main'
}
```
[Install or upgrade the Claude SDK](https://platform.claude.com/docs/en/manage-claude/opentelemetry/sdk#install-the-sdk) to a version that emits spans. The SDK emits a span for each request and sends the trace context that joins the API's spans to yours.
`ARIZE_PROJECT_NAME` must match the destination's **Project name**, so the two sources of spans meet in one project:
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
export ARIZE_SPACE_ID=""
export ARIZE_API_KEY=""
export ARIZE_PROJECT_NAME="claude-managed-agents-triage"
export ANTHROPIC_API_KEY=""
export ANTHROPIC_OPEN_TELEMETRY_TRACES_CONTENT_MODE="content"
```
`ANTHROPIC_OPEN_TELEMETRY_TRACES_CONTENT_MODE=content` makes the SDK's spans carry prompts and responses, matching the **Full content** mode of the destination. The programs send to the US endpoint by default. For a space in another region, also set `ARIZE_COLLECTOR_ENDPOINT` to that region's endpoint, for example `https://otlp.eu-west-1a.arize.com/v1` for EU spaces.
For Python and TypeScript, register the tracer provider in its own module, so that it runs before the Claude SDK makes its first request:
```python Python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
# instrumentation.py
from arize.otel import register
# register() reads ARIZE_SPACE_ID, ARIZE_API_KEY, ARIZE_PROJECT_NAME, and
# ARIZE_COLLECTOR_ENDPOINT from the environment.
tracer_provider = register()
print("Arize AX tracing initialized for Claude Managed Agents.")
```
```typescript TypeScript theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
// instrumentation.ts
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
import {
SEMRESATTRS_PROJECT_NAME,
} from "@arizeai/openinference-semantic-conventions";
const projectName = process.env.ARIZE_PROJECT_NAME ?? "default";
// For EU or Canada spaces, set ARIZE_COLLECTOR_ENDPOINT to your region's
// endpoint, such as https://otlp.eu-west-1a.arize.com/v1.
const endpoint =
process.env.ARIZE_COLLECTOR_ENDPOINT ?? "https://otlp.arize.com/v1";
export const provider = new NodeTracerProvider({
resource: resourceFromAttributes({
[ATTR_SERVICE_NAME]: projectName,
[SEMRESATTRS_PROJECT_NAME]: projectName,
}),
spanProcessors: [
new BatchSpanProcessor(
new OTLPTraceExporter({
url: `${endpoint}/traces`,
headers: {
"arize-api-key": process.env.ARIZE_API_KEY ?? "",
"arize-space-id": process.env.ARIZE_SPACE_ID ?? "",
},
}),
),
],
});
// register() installs the W3C trace context propagator and the context
// manager, so the SDK's spans nest under your own spans.
provider.register();
console.log("Arize AX tracing initialized for Claude Managed Agents.");
```
For Go and Java, the same setup is at the top of `main` in [Build the triage agent](#build-the-triage-agent). It also registers the W3C trace context propagator, which neither language registers by default. The SDK uses the propagator to send each span's trace context to the Claude API, which then nests its own spans under yours.
## Build the triage agent
The program has three functions:
* **`create_agent`** creates an environment and an agent. The agent has Anthropic's built-in toolset, with the web tools turned off, and one custom tool, `list_issues`. A custom tool runs in your application, not in the sandbox. That makes it the right place for anything that needs your data or your credentials.
* **`list_issues`** answers the custom tool from a small list of sample issues. It wraps that work in its own `TOOL` span, with OpenInference's `@tracer.tool` decorator in Python and its `traceTool` wrapper in TypeScript. In Python, `@tracer.tool` reads the tool's name, description, and parameters from the function's name, docstring, and signature. TypeScript, Go, and Java set them on the span themselves. The Claude API sees the call and the result but not your code, so this span is the only record of what your application did. If you run custom tools with the Claude SDK's session tool runner instead, the SDK creates this `TOOL` span for you, so don't add your own.
* **`run_triage`** starts a session inside a `triage-request` `CHAIN` span, created with `@tracer.chain` in Python and `traceChain` in TypeScript, sends the request, and reads the session's event stream. When the agent calls `list_issues`, the session pauses until your application sends back the result. The loop ends when the session goes idle for any reason other than `requires_action`, which means it's waiting on you.
The `triage-request` span records the request as its input and the agent's answer as its output. The Python and TypeScript helpers record them from the function's argument and return value, and Go and Java set the `input.value` and `output.value` attributes themselves. The evaluator in [Evaluate the triage](#evaluate-the-triage) scores those two values. Every span you create also sets its status: `OK` on success, or an error with the exception if a request fails.
```python Python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
# triage.py
# Importing instrumentation first registers the tracer provider
# before the Claude SDK makes its first request.
from instrumentation import tracer_provider
import json
import anthropic
ISSUES = [
{"number": 101, "title": "Login fails with SSO for all EU users",
"labels": ["bug", "auth"], "comments": 14},
{"number": 102, "title": "Typo in the README install section",
"labels": ["docs"], "comments": 1},
{"number": 103, "title": "Export to CSV drops the last row",
"labels": ["bug", "export"], "comments": 6},
{"number": 104, "title": "Add dark mode to the settings page",
"labels": ["enhancement"], "comments": 9},
]
REQUEST = (
"Triage the open issues. Use bash to count the open issues for each "
"label, then name the single most urgent issue and explain why in "
"two sentences."
)
client = anthropic.Anthropic()
tracer = tracer_provider.get_tracer("issue-triage")
def create_agent():
environment = client.beta.environments.create(
name="issue-triage",
config={"type": "cloud", "networking": {"type": "limited"}},
)
agent = client.beta.agents.create(
name="Issue Triage Agent",
model="claude-opus-5-5",
system=(
"You triage GitHub issues for a small open-source project. "
"Call list_issues to read the issues. Use bash for any "
"counting or sorting."
),
tools=[
{
"type": "agent_toolset_20260401",
"configs": [
{"name": "web_fetch", "enabled": False},
{"name": "web_search", "enabled": False},
],
},
{
"type": "custom",
"name": "list_issues",
"description": "List the repository's issues as JSON.",
"input_schema": {
"type": "object",
"properties": {
"state": {"type": "string", "enum": ["open"]},
},
"required": ["state"],
},
},
],
)
return environment, agent
# Your own span for the tool your application runs. The Claude API
# can't see inside this function, so nothing else would trace it.
@tracer.tool(name="list_issues")
def list_issues(state: str) -> str:
"""List the repository's issues as JSON."""
return json.dumps(ISSUES)
@tracer.chain(name="triage-request")
def run_triage(request: str) -> str:
session = client.beta.sessions.create(
agent={"type": "agent", "id": agent.id, "version": agent.version},
environment_id=environment.id,
)
replies = []
# Open the stream before sending the message so no events are
# missed.
with client.beta.sessions.events.stream(
session_id=session.id,
) as stream:
client.beta.sessions.events.send(
session_id=session.id,
events=[
{
"type": "user.message",
"content": [{"type": "text", "text": request}],
},
],
)
for event in stream:
if event.type == "agent.message":
for block in event.content:
if block.type == "text":
replies.append(block.text)
elif event.type == "agent.custom_tool_use":
client.beta.sessions.events.send(
session_id=session.id,
events=[
{
"type": "user.custom_tool_result",
"custom_tool_use_id": event.id,
"content": [
{
"type": "text",
"text": list_issues(**event.input),
},
],
},
],
)
elif event.type == "session.status_idle":
if event.stop_reason.type != "requires_action":
break
elif event.type == "session.status_terminated":
break
return "\n".join(replies)
environment, agent = create_agent()
print(run_triage(REQUEST))
tracer_provider.shutdown()
```
```typescript TypeScript theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
// triage.ts
// Importing instrumentation first registers the tracer provider
// before the Claude SDK makes its first request.
import { provider } from "./instrumentation";
import Anthropic from "@anthropic-ai/sdk";
import { traceChain, traceTool } from "@arizeai/openinference-core";
const ISSUES = [
{ number: 101, title: "Login fails with SSO for all EU users",
labels: ["bug", "auth"], comments: 14 },
{ number: 102, title: "Typo in the README install section",
labels: ["docs"], comments: 1 },
{ number: 103, title: "Export to CSV drops the last row",
labels: ["bug", "export"], comments: 6 },
{ number: 104, title: "Add dark mode to the settings page",
labels: ["enhancement"], comments: 9 },
];
const LIST_ISSUES_PARAMETERS = {
type: "object" as const,
properties: { state: { type: "string", enum: ["open"] } },
required: ["state"],
};
const REQUEST =
"Triage the open issues. Use bash to count the open issues for each " +
"label, then name the single most urgent issue and explain why in " +
"two sentences.";
const client = new Anthropic();
async function createAgent() {
const environment = await client.beta.environments.create({
name: "issue-triage",
config: { type: "cloud", networking: { type: "limited" } },
});
const agent = await client.beta.agents.create({
name: "Issue Triage Agent",
model: "claude-opus-5-5",
system:
"You triage GitHub issues for a small open-source project. " +
"Call list_issues to read the issues. Use bash for any " +
"counting or sorting.",
tools: [
{
type: "agent_toolset_20260401",
configs: [
{ name: "web_fetch", enabled: false },
{ name: "web_search", enabled: false },
],
},
{
type: "custom",
name: "list_issues",
description: "List the repository's issues as JSON.",
input_schema: LIST_ISSUES_PARAMETERS,
},
],
});
return { environment, agent };
}
// Your own span for the tool your application runs. The Claude API
// can't see inside this function, so nothing else would trace it.
const listIssues = traceTool(
(input: unknown): string => JSON.stringify(ISSUES),
{
name: "list_issues",
attributes: {
"tool.name": "list_issues",
"tool.description": "List the repository's issues as JSON.",
"tool.parameters": JSON.stringify(LIST_ISSUES_PARAMETERS),
},
},
);
function traceTriage(environmentId: string, agentId: string,
agentVersion: number) {
return traceChain(async (request: string): Promise => {
const session = await client.beta.sessions.create({
agent: { type: "agent", id: agentId, version: agentVersion },
environment_id: environmentId,
});
const replies: string[] = [];
// Open the stream before sending the message so no events are missed.
const stream = await client.beta.sessions.events.stream(session.id);
await client.beta.sessions.events.send(session.id, {
events: [
{ type: "user.message", content: [{ type: "text", text: request }] },
],
});
loop: for await (const event of stream) {
switch (event.type) {
case "agent.message":
for (const block of event.content) {
if (block.type === "text") replies.push(block.text);
}
break;
case "agent.custom_tool_use":
await client.beta.sessions.events.send(session.id, {
events: [
{
type: "user.custom_tool_result",
custom_tool_use_id: event.id,
content: [{ type: "text", text: listIssues(event.input) }],
},
],
});
break;
case "session.status_idle":
if (event.stop_reason.type !== "requires_action") break loop;
break;
case "session.status_terminated":
break loop;
}
}
return replies.join("\n");
}, { name: "triage-request" });
}
async function main() {
const { environment, agent } = await createAgent();
const runTriage = traceTriage(environment.id, agent.id, agent.version);
console.log(await runTriage(REQUEST));
await provider.shutdown();
}
main().catch((err) => {
console.error(err);
process.exit(1);
});
```
```go Go theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
// main.go
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"strings"
"time"
arizeotel "github.com/Arize-ai/arize-otel-go"
"github.com/anthropics/anthropic-sdk-go"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/trace"
)
type issue struct {
Number int `json:"number"`
Title string `json:"title"`
Labels []string `json:"labels"`
Comments int `json:"comments"`
}
var issues = []issue{
{101, "Login fails with SSO for all EU users",
[]string{"bug", "auth"}, 14},
{102, "Typo in the README install section", []string{"docs"}, 1},
{103, "Export to CSV drops the last row",
[]string{"bug", "export"}, 6},
{104, "Add dark mode to the settings page",
[]string{"enhancement"}, 9},
}
const request = "Triage the open issues. Use bash to count the open " +
"issues for each label, then name the single most urgent issue " +
"and explain why in two sentences."
var tracer = otel.Tracer("issue-triage")
// listIssuesParameters is the list_issues input schema, recorded on its
// TOOL span.
const listIssuesParameters = `{"type":"object",` +
`"properties":{"state":{"type":"string","enum":["open"]}},` +
`"required":["state"]}`
func main() {
if err := run(context.Background()); err != nil {
log.Fatal(err)
}
}
// run returns errors instead of exiting, so the deferred Shutdown flushes
// the spans to Arize AX even when a request fails.
func run(ctx context.Context) error {
// Register reads ARIZE_SPACE_ID, ARIZE_API_KEY, and
// ARIZE_COLLECTOR_ENDPOINT from the environment.
tp, err := arizeotel.Register(ctx, arizeotel.Options{
ProjectName: os.Getenv("ARIZE_PROJECT_NAME"),
})
if err != nil {
return fmt.Errorf("register tracer: %w", err)
}
defer func() {
shutdownCtx, cancel := context.WithTimeout(
context.Background(), 5*time.Second)
defer cancel()
_ = tp.Shutdown(shutdownCtx)
}()
// OpenTelemetry for Go registers no propagator by default. The Claude
// SDK needs W3C trace context to link its spans to the API's spans.
otel.SetTextMapPropagator(propagation.TraceContext{})
client := anthropic.NewClient()
environmentID, agentID, agentVersion, err := createAgent(ctx, client)
if err != nil {
return fmt.Errorf("create agent: %w", err)
}
answer, err := runTriage(ctx, client, environmentID, agentID,
agentVersion)
if err != nil {
return fmt.Errorf("run triage: %w", err)
}
fmt.Println(answer)
return nil
}
func createAgent(ctx context.Context, client anthropic.Client) (
string, string, int64, error) {
environment, err := client.Beta.Environments.New(ctx,
anthropic.BetaEnvironmentNewParams{
Name: "issue-triage",
Config: anthropic.BetaEnvironmentNewParamsConfigUnion{
OfCloud: &anthropic.BetaCloudConfigParams{
Networking: anthropic.BetaCloudConfigParamsNetworkingUnion{
OfLimited: &anthropic.BetaLimitedNetworkParams{},
},
},
},
})
if err != nil {
return "", "", 0, err
}
listIssuesTool := anthropic.BetaManagedAgentsCustomToolParams{
Type: anthropic.BetaManagedAgentsCustomToolParamsTypeCustom,
Name: "list_issues",
Description: "List the repository's issues as JSON.",
}
listIssuesTool.InputSchema.Properties = map[string]any{
"state": map[string]any{"type": "string", "enum": []string{"open"}},
}
listIssuesTool.InputSchema.Required = []string{"state"}
agent, err := client.Beta.Agents.New(ctx, anthropic.BetaAgentNewParams{
Name: "Issue Triage Agent",
Model: anthropic.BetaManagedAgentsModelConfigParams{
ID: "claude-opus-5-5",
},
System: anthropic.String(
"You triage GitHub issues for a small open-source project. " +
"Call list_issues to read the issues. Use bash for any " +
"counting or sorting."),
Tools: []anthropic.BetaAgentNewParamsToolUnion{
{OfAgentToolset20260401: toolset()},
{OfCustom: &listIssuesTool},
},
})
if err != nil {
return "", "", 0, err
}
return environment.ID, agent.ID, agent.Version, nil
}
// listIssues records your own span for the tool your application runs.
// The Claude API can't see inside this function.
func listIssues(ctx context.Context, input map[string]any) string {
_, span := tracer.Start(ctx, "list_issues")
defer span.End()
inputJSON, _ := json.Marshal(input)
result, _ := json.Marshal(issues)
span.SetAttributes(
attribute.String("openinference.span.kind", "TOOL"),
attribute.String("tool.name", "list_issues"),
attribute.String("tool.description",
"List the repository's issues as JSON."),
attribute.String("tool.parameters", listIssuesParameters),
attribute.String("input.value", string(inputJSON)),
attribute.String("output.value", string(result)),
)
span.SetStatus(codes.Ok, "")
return string(result)
}
func runTriage(ctx context.Context, client anthropic.Client,
environmentID, agentID string,
agentVersion int64) (answer string, err error) {
ctx, root := tracer.Start(ctx, "triage-request", trace.WithAttributes(
attribute.String("openinference.span.kind", "CHAIN"),
attribute.String("input.value", request),
))
defer func() {
if err != nil {
root.RecordError(err)
root.SetStatus(codes.Error, err.Error())
} else {
root.SetStatus(codes.Ok, "")
}
root.End()
}()
sessionParams := anthropic.BetaSessionNewParams{
Agent: anthropic.BetaSessionNewParamsAgentUnion{
OfBetaManagedAgentsAgents: &anthropic.BetaManagedAgentsAgentParams{
Type: anthropic.BetaManagedAgentsAgentParamsTypeAgent,
ID: agentID,
Version: anthropic.Int(agentVersion),
},
},
EnvironmentID: environmentID,
}
session, err := client.Beta.Sessions.New(ctx, sessionParams)
if err != nil {
return "", err
}
// Open the stream before sending the message so no events are missed.
stream := client.Beta.Sessions.Events.StreamEvents(ctx, session.ID,
anthropic.BetaSessionEventStreamParams{})
defer stream.Close()
if err := send(ctx, client, session.ID,
anthropic.BetaManagedAgentsEventParamsUnion{
OfUserMessage: &anthropic.BetaManagedAgentsUserMessageEventParams{
Type: anthropic.
BetaManagedAgentsUserMessageEventParamsTypeUserMessage,
Content: []anthropic.
BetaManagedAgentsUserMessageEventParamsContentUnion{{
OfText: textBlock(request),
}},
},
}); err != nil {
return "", err
}
var replies []string
events:
for stream.Next() {
switch event := stream.Current().AsAny().(type) {
case anthropic.BetaManagedAgentsAgentMessageEvent:
for _, block := range event.Content {
replies = append(replies, block.Text)
}
case anthropic.BetaManagedAgentsAgentCustomToolUseEvent:
result := toolResult(event.ID, listIssues(ctx, event.Input))
if err := send(ctx, client, session.ID, result); err != nil {
return "", err
}
case anthropic.BetaManagedAgentsSessionStatusIdleEvent:
if event.StopReason.Type != "requires_action" {
break events
}
case anthropic.BetaManagedAgentsSessionStatusTerminatedEvent:
break events
}
}
if err := stream.Err(); err != nil {
return "", err
}
answer = strings.Join(replies, "\n")
root.SetAttributes(attribute.String("output.value", answer))
return answer, nil
}
func send(ctx context.Context, client anthropic.Client, sessionID string,
event anthropic.BetaManagedAgentsEventParamsUnion) error {
_, err := client.Beta.Sessions.Events.Send(ctx, sessionID,
anthropic.BetaSessionEventSendParams{
Events: []anthropic.BetaManagedAgentsEventParamsUnion{event},
})
return err
}
// toolResult answers a custom tool call. The SDK's
// BetaManagedAgentsEventParamsOfUserCustomToolResult helper leaves the
// event type unset, which the API rejects.
func toolResult(id, text string) anthropic.BetaManagedAgentsEventParamsUnion {
var result anthropic.BetaManagedAgentsUserCustomToolResultEventParams
result.Type = anthropic.
BetaManagedAgentsUserCustomToolResultEventParamsTypeUserCustomToolResult
result.CustomToolUseID = id
result.Content = []anthropic.
BetaManagedAgentsUserCustomToolResultEventParamsContentUnion{{
OfText: textBlock(text),
}}
return anthropic.BetaManagedAgentsEventParamsUnion{
OfUserCustomToolResult: &result,
}
}
func textBlock(text string) *anthropic.BetaManagedAgentsTextBlockParam {
return &anthropic.BetaManagedAgentsTextBlockParam{
Type: anthropic.BetaManagedAgentsTextBlockTypeText,
Text: text,
}
}
func toolset() *anthropic.BetaManagedAgentsAgentToolset20260401Params {
ts := &anthropic.BetaManagedAgentsAgentToolset20260401Params{
Configs: []anthropic.BetaManagedAgentsAgentToolConfigParamsUnion{
{OfWebFetch: &anthropic.BetaManagedAgentsWebFetchToolConfigParams{
Enabled: anthropic.Bool(false),
}},
{OfWebSearch: &anthropic.BetaManagedAgentsWebSearchToolConfigParams{
Enabled: anthropic.Bool(false),
}},
},
}
ts.Type = anthropic.
BetaManagedAgentsAgentToolset20260401ParamsTypeAgentToolset20260401
return ts
}
```
```java Java theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
// src/main/java/example/Main.java
package example;
import com.anthropic.client.AnthropicClient;
import com.anthropic.client.okhttp.AnthropicOkHttpClient;
import com.anthropic.core.JsonValue;
import com.anthropic.core.ObjectMappers;
import com.anthropic.models.beta.agents.*;
import com.anthropic.models.beta.environments.BetaCloudConfigParams;
import com.anthropic.models.beta.environments.BetaEnvironment;
import com.anthropic.models.beta.environments.BetaLimitedNetworkParams;
import com.anthropic.models.beta.environments.EnvironmentCreateParams;
import com.anthropic.models.beta.sessions.BetaManagedAgentsAgentParams;
import com.anthropic.models.beta.sessions.SessionCreateParams;
import com.anthropic.models.beta.sessions.events.*;
import com.fasterxml.jackson.core.JsonProcessingException;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator;
import io.opentelemetry.context.Scope;
import io.opentelemetry.context.propagation.ContextPropagators;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
import java.net.URI;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
public class Main {
// The list_issues input schema, recorded on its TOOL span.
private static final String LIST_ISSUES_PARAMETERS =
"{\"type\":\"object\",\"properties\":{\"state\":"
+ "{\"type\":\"string\",\"enum\":[\"open\"]}},"
+ "\"required\":[\"state\"]}";
private static final String ISSUES = """
[
{"number": 101, "title": "Login fails with SSO for all EU users",
"labels": ["bug", "auth"], "comments": 14},
{"number": 102, "title": "Typo in the README install section",
"labels": ["docs"], "comments": 1},
{"number": 103, "title": "Export to CSV drops the last row",
"labels": ["bug", "export"], "comments": 6},
{"number": 104, "title": "Add dark mode to the settings page",
"labels": ["enhancement"], "comments": 9}
]
""";
private static final String REQUEST =
"Triage the open issues. Use bash to count the open issues for "
+ "each label, then name the single most urgent issue and "
+ "explain why in two sentences.";
private static AnthropicClient client;
private static Tracer tracer;
public static void main(String[] args) throws Exception {
String project = System.getenv().getOrDefault(
"ARIZE_PROJECT_NAME", "default");
Resource resource = Resource.getDefault().merge(Resource.create(
Attributes.of(
AttributeKey.stringKey("service.name"), project,
AttributeKey.stringKey("openinference.project.name"),
project)));
// For EU or Canada spaces, set ARIZE_COLLECTOR_ENDPOINT to your
// region's endpoint, such as https://otlp.eu-west-1a.arize.com/v1.
URI collector = URI.create(System.getenv().getOrDefault(
"ARIZE_COLLECTOR_ENDPOINT", "https://otlp.arize.com/v1"));
OtlpGrpcSpanExporter exporter = OtlpGrpcSpanExporter.builder()
.setEndpoint("https://" + collector.getHost() + ":443")
.setHeaders(() -> Map.of(
"authorization", System.getenv("ARIZE_API_KEY"),
"arize-space-id", System.getenv("ARIZE_SPACE_ID")))
.build();
SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(exporter).build())
.setResource(resource)
.build();
// The Claude SDK reads the global OpenTelemetry instance. It needs
// the W3C propagator to link its spans to the API's spans.
OpenTelemetrySdk otel = OpenTelemetrySdk.builder()
.setTracerProvider(tracerProvider)
.setPropagators(ContextPropagators.create(
W3CTraceContextPropagator.getInstance()))
.buildAndRegisterGlobal();
client = AnthropicOkHttpClient.fromEnv();
tracer = otel.getTracer("issue-triage");
BetaEnvironment environment = createEnvironment();
BetaManagedAgentsAgent agent = createAgent();
System.out.println(runTriage(environment, agent));
tracerProvider.forceFlush().join(10, TimeUnit.SECONDS);
tracerProvider.shutdown().join(10, TimeUnit.SECONDS);
}
private static BetaEnvironment createEnvironment() {
return client.beta().environments().create(
EnvironmentCreateParams.builder()
.name("issue-triage")
.config(BetaCloudConfigParams.builder()
.networking(BetaLimitedNetworkParams.builder().build())
.build())
.build());
}
private static BetaManagedAgentsAgent createAgent() {
var toolset = BetaManagedAgentsAgentToolset20260401Params.builder()
.type(BetaManagedAgentsAgentToolset20260401Params.Type
.AGENT_TOOLSET_20260401)
.addConfig(BetaManagedAgentsWebFetchToolConfigParams.builder()
.enabled(false).build())
.addConfig(BetaManagedAgentsWebSearchToolConfigParams.builder()
.enabled(false).build())
.build();
var listIssues = BetaManagedAgentsCustomToolParams.builder()
.type(BetaManagedAgentsCustomToolParams.Type.CUSTOM)
.name("list_issues")
.description("List the repository's issues as JSON.")
.inputSchema(BetaManagedAgentsCustomToolInputSchema.builder()
.properties(BetaManagedAgentsCustomToolInputSchema.Properties
.builder()
.putAdditionalProperty("state", JsonValue.from(Map.of(
"type", "string", "enum", List.of("open"))))
.build())
.addRequired("state")
.build())
.build();
return client.beta().agents().create(AgentCreateParams.builder()
.name("Issue Triage Agent")
.model(BetaManagedAgentsModel.of("claude-opus-5-5"))
.system("You triage GitHub issues for a small open-source "
+ "project. Call list_issues to read the issues. Use bash "
+ "for any counting or sorting.")
.addTool(toolset)
.addTool(listIssues)
.build());
}
// Your own span for the tool your application runs. The Claude API
// can't see inside this method, so nothing else would trace it.
private static String listIssues(
BetaManagedAgentsAgentCustomToolUseEvent call)
throws JsonProcessingException {
Span span = tracer.spanBuilder("list_issues").startSpan();
try (Scope scope = span.makeCurrent()) {
span.setAttribute("openinference.span.kind", "TOOL");
span.setAttribute("tool.name", "list_issues");
span.setAttribute("tool.description",
"List the repository's issues as JSON.");
span.setAttribute("tool.parameters", LIST_ISSUES_PARAMETERS);
span.setAttribute("input.value",
ObjectMappers.jsonMapper().writeValueAsString(call.input()));
span.setAttribute("output.value", ISSUES);
span.setStatus(StatusCode.OK);
return ISSUES;
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR, e.getMessage());
throw e;
} finally {
span.end();
}
}
private static BetaManagedAgentsUserCustomToolResultEventParams
toolResult(BetaManagedAgentsAgentCustomToolUseEvent call)
throws JsonProcessingException {
return BetaManagedAgentsUserCustomToolResultEventParams.builder()
.type(BetaManagedAgentsUserCustomToolResultEventParams.Type
.USER_CUSTOM_TOOL_RESULT)
.customToolUseId(call.id())
.addTextContent(listIssues(call))
.build();
}
private static String runTriage(BetaEnvironment environment,
BetaManagedAgentsAgent agent) throws JsonProcessingException {
Span root = tracer.spanBuilder("triage-request").startSpan();
try (Scope scope = root.makeCurrent()) {
root.setAttribute("openinference.span.kind", "CHAIN");
root.setAttribute("input.value", REQUEST);
var session = client.beta().sessions().create(
SessionCreateParams.builder()
.agent(BetaManagedAgentsAgentParams.builder()
.type(BetaManagedAgentsAgentParams.Type.AGENT)
.id(agent.id())
.version(agent.version())
.build())
.environmentId(environment.id())
.build());
var events = client.beta().sessions().events();
List replies = new ArrayList<>();
// Open the stream before sending the message so no events
// are missed.
try (var stream = events.streamStreaming(session.id())) {
events.send(session.id(), EventSendParams.builder()
.addEvent(BetaManagedAgentsUserMessageEventParams.builder()
.type(BetaManagedAgentsUserMessageEventParams.Type
.USER_MESSAGE)
.addTextContent(REQUEST)
.build())
.build());
Iterable received =
stream.stream()::iterator;
for (var event : received) {
if (event.isAgentMessage()) {
event.asAgentMessage().content()
.forEach(block -> block.text()
.ifPresent(text -> replies.add(text.text())));
} else if (event.isAgentCustomToolUse()) {
var call = event.asAgentCustomToolUse();
events.send(session.id(), EventSendParams.builder()
.addEvent(toolResult(call))
.build());
} else if (event.isSessionStatusIdle()) {
if (!event.asSessionStatusIdle().stopReason()
.isRequiresAction()) {
break;
}
} else if (event.isSessionStatusTerminated()) {
break;
}
}
}
String answer = String.join("\n", replies);
root.setAttribute("output.value", answer);
root.setStatus(StatusCode.OK);
return answer;
} catch (Exception e) {
root.recordException(e);
root.setStatus(StatusCode.ERROR, e.getMessage());
throw e;
} finally {
root.end();
}
}
}
```
Agents and environments are persistent and versioned. The program creates them on each run to stay self-contained. In your own application, create them once, store their IDs, and pass the IDs to each `sessions.create` call.
## Run a session
Run the program:
```bash Python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
python triage.py
```
```bash TypeScript theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
npx tsx triage.ts
```
```bash Go theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
go run main.go
```
```bash Java theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
gradle run
```
The agent calls `list_issues`, counts the issues by label with bash in its sandbox, and replies. The wording and layout vary from run to run, but the output looks similar to this:
```text wrap theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
There are 4 open issues. Here's how many are open under each label, counted with bash:
| Label | Open issues |
|---|---|
| bug | 2 (#101, #103) |
| auth | 1 (#101) |
| docs | 1 (#102) |
| enhancement | 1 (#104) |
| export | 1 (#103) |
The totals add up to more than 4 because #101 and #103 each have two labels.
**Most urgent: #101, "Login fails with SSO for all EU users."** It's a bug that stops a whole region of users from logging in at all, which makes it worse than #103's lost CSV row or the docs and feature requests. It also has the most discussion of any open issue (14 comments), so it's clearly affecting many people and should be fixed first.
```
## Explore the trace
Open your Arize AX space and select the **`claude-managed-agents-triage`** project. A new trace appears within about a minute. The API's spans can arrive after yours. Open the trace to see the session from both sides:
```text theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
triage-request CHAIN, your application
├─ anthropic.sessions.create SDK, with the API's span inside
├─ anthropic.sessions.events.stream SDK, with the API's span inside
├─ anthropic.sessions.events.send SDK, with the API's span inside
│ └─ anthropic.session.turn AGENT, stop reason requires_action
│ └─ anthropic.model_request LLM
├─ list_issues TOOL, your application
└─ anthropic.sessions.events.send SDK, with the API's span inside
└─ anthropic.session.turn AGENT, stop reason end_turn
├─ anthropic.model_request LLM
├─ anthropic.tool_use bash TOOL, in the sandbox
└─ anthropic.model_request LLM
```
* **`triage-request`** is your root span, with the request as input and the agent's answer as output.
* Under it are the SDK's `CLIENT` spans, one for each request your application sent. Each one contains the API's span for the same request, with the same name.
* Under the request that sent your message is the agent's first turn. It ends with the stop reason `requires_action` when the agent calls `list_issues`. Your `list_issues` `TOOL` span follows, then the request that sent the result, and a second turn in which the agent runs bash and answers.
* The SDK's spans for creating the environment and the agent are in traces of their own, because the program creates them before it starts `triage-request`.
Open an `anthropic.session.turn` span to read the system prompt and the turn's input and output messages, including the tool calls and their results. In the first turn, the agent's output is the `list_issues` call:
Open an `anthropic.model_request` span to see the model and token usage of one model request. Open the `bash` tool span to see the command that the agent ran in its sandbox. Its arguments are the span's input:
The command's result is in the span's `anthropic.tool_result.content` attribute, on the **Attributes** tab:
The API's spans also carry the session's thread ID in `gen_ai.conversation.id`, and Arize AX uses it as the session ID, so each thread appears in the project's **Sessions** tab. Open **Sessions** and select the session to read the thread as a conversation.
The Sessions list shows no first message, last message, or duration for these sessions. Arize AX fills in those columns from the root span of each trace in a session, and the root here is your `triage-request` span. The session ID is only on the SDK's and the API's spans further down the trace, so the list has nothing to summarize. Selecting the session still shows the whole conversation.
A Managed Agents session can have more than one [thread](https://platform.claude.com/docs/en/managed-agents/session-threads), for example when its agent hands work to other agents. Each thread is a separate session in Arize AX, but every span in every thread carries the same `anthropic.session.id`. To see all of a session's work in one place, filter the project's spans on `anthropic.session.id`. A turn that starts because an agent on another thread replied or finished has no parent, so it's the root of a trace of its own.
## Evaluate the triage
The trace shows what the agent did. An evaluator tells you whether it did it well. Sample issue #101 locks out a whole region, so a good triage names it as the most urgent. Create an LLM-as-a-judge evaluator that checks the agent's answer against that:
In Arize AX, open **Evaluators** and click **New Task**. Name the task `triage-quality` and select the `claude-managed-agents-triage` project.
Add a filter on the span name `triage-request`, so the task scores your root spans.
Click **Add Evaluator**, choose **Create Custom**, and use this template:
```text wrap theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
You are evaluating an issue-triage agent. It was asked to pick the single most urgent open issue and explain why.
[Request]: {input}
[Answer]: {output}
An urgent issue blocks many users from core functionality. A good answer names the issue that blocks the most users and gives a reason based on its impact.
Respond with "correct" if the answer picks the issue with the widest impact and justifies it, or "incorrect" otherwise.
```
Map `input` to `attributes.input.value` and `output` to `attributes.output.value`, and set the choices to `correct` (score 1) and `incorrect` (score 0).
Turn on **Run continuously**, and run it once on your existing traces as well. Then click **Create Task**.
Run the program a few more times. Each `triage-request` span gets a `triage-quality` label.
Filter the project on `incorrect` to find the sessions where the agent got it wrong, and open their traces to see whether the cause was the tool result, the bash step, or the model's reasoning. For the other ways to create a task, such as with Arize Skills or Alyx, and for the full set of options, see [Run evals on traces](/docs/ax/evaluate/run-evals-on-traces).
## Summary
You traced a Claude Managed Agents session from both sides and evaluated its result:
* A Claude Console destination sends the spans that the Claude API creates to an Arize AX project, routed by the destination's project name.
* The Claude SDK's spans and W3C trace context join those spans to your application's own trace.
* Your own `TOOL` span records the custom tool that your application runs, which the Claude API can't see.
* An LLM-as-a-judge task scores each triage, so you can find the runs that went wrong and trace their cause.
## Next steps
The reference for the Console destination, the SDK setup, and the spans Arize AX captures.
Score whole conversations instead of single answers.
Score tool selection, tool results, and goal completion.
How Arize AX converts `gen_ai.*` spans to OpenInference.