> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Manual Instrumentation
> Create spans explicitly using the OpenTelemetry API for custom logic, unsupported frameworks, or fine-grained control
Auto-instrumentation covers supported frameworks. For everything else — custom logic, tool execution, unsupported frameworks, or fine-grained control — you create spans explicitly using the OpenTelemetry API with [OpenInference Semantic Conventions](https://github.com/Arize-ai/openinference).
**AI agent requirement.** If you are using Codex, Cursor, Claude Code, or another coding agent, install and consult the Arize [`arize-instrumentation` skill](/docs/ax/set-up-with-ai-assistants#skills) **before modifying application code**. It detects your stack and automatically wires up the right OpenInference setup for your use case — teams that instrument with our skills succeed far more often than agents left to infer the setup from docs or intuition.
```bash theme={null}
npx skills add Arize-ai/arize-skills --skill "arize-instrumentation" --yes
```
Use the code steps below as a fallback only if skill installation fails or your environment cannot install skills. [Learn more about coding-agent setup →](/docs/ax/set-up-with-ai-assistants)
# Set Up with Skills or Code
**AI agent requirement.** If you are using Codex, Cursor, Claude Code, or another coding agent, install and consult the Arize [`arize-instrumentation` skill](/docs/ax/set-up-with-ai-assistants#skills) **before modifying application code**. It detects your stack and automatically wires up the right OpenInference setup for your use case — teams that instrument with our skills succeed far more often than agents left to infer the setup from docs or intuition.
```bash theme={null}
npx skills add Arize-ai/arize-skills --skill "arize-instrumentation" --yes
```
Use the code steps below as a fallback only if skill installation fails or your environment cannot install skills. [Learn more about coding-agent setup →](/docs/ax/set-up-with-ai-assistants)
Three steps to add manual spans with your AI coding agent:
**Install skill**
```bash theme={null}
npx skills add Arize-ai/arize-skills --skill "arize-instrumentation" --yes
```
**Set up authentication**
```bash theme={null}
export ARIZE_API_KEY="YOUR_API_KEY"
export ARIZE_SPACE_ID="YOUR_SPACE_ID"
```
**Ask it to add manual spans**
```bash theme={null}
# Ask your AI coding agent:
"Wrap my custom tool functions with manual spans using OpenInference conventions"
```
Works with Cursor, Claude Code, Codex, and more. The skill picks the right span kinds, sets the right OpenInference attributes, and handles context propagation for you:
Set up the OpenTelemetry SDK, register a tracer provider with your Arize credentials, and create spans with [OpenInference](https://github.com/Arize-ai/openinference) attributes.
```bash theme={null}
pip install openinference-instrumentation opentelemetry-api opentelemetry-sdk arize-otel
```
```bash theme={null}
npm install @opentelemetry/api @opentelemetry/exporter-trace-otlp-proto @opentelemetry/resources @opentelemetry/sdk-trace-base @opentelemetry/sdk-trace-node @opentelemetry/semantic-conventions @arizeai/openinference-semantic-conventions openai
```
Requires Go 1.25+. Install [`arize-otel-go`](https://github.com/Arize-ai/arize-otel-go) for tracer setup, plus the OpenInference [`semantic-conventions`](https://github.com/Arize-ai/openinference/tree/main/go/openinference-semantic-conventions) and [`instrumentation`](https://github.com/Arize-ai/openinference/tree/main/go/openinference-instrumentation) packages — the first for typed attribute-key constants, the second for context-scoped helpers (`WithSession`, `WithUser`, `WithMetadata`, `WithTags`, `WithSuppression`):
```bash theme={null}
go get \
github.com/Arize-ai/arize-otel-go \
github.com/Arize-ai/openinference/go/openinference-instrumentation \
github.com/Arize-ai/openinference/go/openinference-semantic-conventions
```
Go to **Settings > API keys** to create your API key and to get the Space ID of your current space.
```python theme={null}
from arize.otel import register
tracer_provider = register(
space_id="YOUR_SPACE_ID",
api_key="YOUR_API_KEY",
project_name="YOUR_PROJECT_NAME",
)
tracer = tracer_provider.get_tracer(__name__)
```
You can also use environment variables `ARIZE_SPACE_ID`, `ARIZE_API_KEY`, and `ARIZE_PROJECT_NAME` instead of passing them directly.
```typescript theme={null}
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { SimpleSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
import { SEMRESATTRS_PROJECT_NAME } from "@arizeai/openinference-semantic-conventions";
const COLLECTOR_ENDPOINT = "https://otlp.arize.com";
const SERVICE_NAME = "manual-js-app";
const provider = new NodeTracerProvider({
resource: resourceFromAttributes({
[ATTR_SERVICE_NAME]: SERVICE_NAME,
[SEMRESATTRS_PROJECT_NAME]: SERVICE_NAME,
}),
spanProcessors: [
new SimpleSpanProcessor(
new OTLPTraceExporter({
url: `${COLLECTOR_ENDPOINT}/v1/traces`,
headers: {
'arize-space-id': 'your-arize-space-id',
'arize-api-key': 'your-arize-api-key',
},
})
),
],
});
provider.register();
```
**Header names differ by transport.** The HTTP/OTLP endpoint (`https://otlp.arize.com/v1/traces`, used by all JS/TS exporters) expects the hyphenated headers **`arize-space-id`** and **`arize-api-key`**. The Python `register()` helper and the OTel Collector use gRPC, where the metadata keys are the unprefixed **`space_id`** and **`api_key`**. Using the wrong form fails silently with no spans — match the form to your transport.
`arizeotel.Register` returns a configured `*sdktrace.TracerProvider`, installs it as the global, sets the required `openinference.project.name` resource attribute, and reads `ARIZE_SPACE_ID` / `ARIZE_API_KEY` / `ARIZE_PROJECT_NAME` from the environment when the matching `Options` fields are unset.
```go theme={null}
package main
import (
"context"
"log"
"time"
arizeotel "github.com/Arize-ai/arize-otel-go"
"go.opentelemetry.io/otel"
)
func main() {
ctx := context.Background()
tp, err := arizeotel.Register(ctx, arizeotel.Options{
ProjectName: "manual-go-app",
})
if err != nil {
log.Printf("register tracer: %v", err)
return
}
defer func() {
// Never call log.Fatalf / os.Exit after a span starts — they
// skip this deferred Shutdown and in-flight spans never flush.
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_ = tp.Shutdown(shutdownCtx)
}()
tracer := otel.Tracer("manual-go-app")
_ = tracer
// ... your app ...
}
```
For EU-region spaces, pass `Endpoint: arizeotel.EndpointArizeEurope`. Set `ARIZE_SPACE_ID` and `ARIZE_API_KEY` as environment variables — never embed credentials in source.
Use your tracer to create spans — either as a context manager (to trace a specific block) or inline.
```python theme={null}
import openai
from opentelemetry.trace import Status, StatusCode
client = openai.OpenAI()
def run_agent(user_input: str) -> str:
with tracer.start_as_current_span("run-agent") as span:
span.set_attribute("openinference.span.kind", "CHAIN")
span.set_attribute("input.value", user_input)
response = call_llm(user_input)
span.set_attribute("output.value", response)
span.set_status(Status(StatusCode.OK))
return response
def call_llm(prompt: str) -> str:
with tracer.start_as_current_span("llm-completion") as span:
span.set_attribute("openinference.span.kind", "LLM")
span.set_attribute("input.value", prompt)
span.set_attribute("llm.model_name", "gpt-5.5")
completion = client.chat.completions.create(
model="gpt-5.5",
messages=[{"role": "user", "content": prompt}],
)
result = completion.choices[0].message.content or ""
span.set_attribute("output.value", result)
span.set_status(Status(StatusCode.OK))
return result
```
```typescript theme={null}
import { trace, SpanStatusCode } from "@opentelemetry/api";
import { INPUT_VALUE, OUTPUT_VALUE, LLM_MODEL_NAME, SemanticConventions,
OpenInferenceSpanKind
} from "@arizeai/openinference-semantic-conventions";
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const tracer = trace.getTracer(SERVICE_NAME);
async function callLLM(prompt: string): Promise {
return tracer.startActiveSpan("call-llm", async (span) => {
span.setAttribute(SemanticConventions.OPENINFERENCE_SPAN_KIND, OpenInferenceSpanKind.LLM);
span.setAttribute(INPUT_VALUE, prompt);
span.setAttribute(LLM_MODEL_NAME, "gpt-5.5");
const response = await openai.chat.completions.create({
model: "gpt-5.5",
messages: [{ role: "user", content: prompt }]
});
const result = response.choices[0].message.content || "";
span.setAttribute(OUTPUT_VALUE, result);
span.setStatus({ code: SpanStatusCode.OK });
span.end();
return result;
});
}
```
Use the [`openinference-semantic-conventions`](https://github.com/Arize-ai/openinference/tree/main/go/openinference-semantic-conventions) package for typed attribute-key constants — no raw strings. The example below shows the fully-manual pattern (CHAIN + LLM) for clients without a first-party instrumentor. If you're using `openai/openai-go` or `anthropics/anthropic-sdk-go`, drop the inner `llm-completion` span — the middleware emits it for you — and keep only the surrounding `CHAIN`.
```go theme={null}
package main
import (
"context"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
semconv "github.com/Arize-ai/openinference/go/openinference-semantic-conventions"
)
var tracer = otel.Tracer("manual-go-app")
func runAgent(ctx context.Context, userInput string) (string, error) {
ctx, span := tracer.Start(ctx, "run-agent")
defer span.End()
span.SetAttributes(
attribute.String(semconv.OpenInferenceSpanKind, semconv.SpanKindChain),
attribute.String(semconv.InputValue, userInput),
)
response, err := callLLM(ctx, userInput)
if err != nil {
span.SetStatus(codes.Error, err.Error())
return "", err
}
span.SetAttributes(attribute.String(semconv.OutputValue, response))
span.SetStatus(codes.Ok, "")
return response, nil
}
func callLLM(ctx context.Context, prompt string) (string, error) {
ctx, span := tracer.Start(ctx, "llm-completion")
defer span.End()
span.SetAttributes(
attribute.String(semconv.OpenInferenceSpanKind, semconv.SpanKindLLM),
attribute.String(semconv.InputValue, prompt),
attribute.String(semconv.LLMModelName, "gpt-5.5"),
)
// ... call your LLM client, then set the output ...
result := "..."
span.SetAttributes(attribute.String(semconv.OutputValue, result))
span.SetStatus(codes.Ok, "")
return result, nil
}
```
Indexed attributes — e.g. message lists — are produced by helper functions: `semconv.LLMInputMessageRoleKey(0)`, `semconv.LLMInputMessageContentKey(0)`, and so on. See the [package reference](https://pkg.go.dev/github.com/Arize-ai/openinference/go/openinference-semantic-conventions) for the full set.
## Use Decorators
Decorators are an alternative to `start_as_current_span` blocks: wrap a function and the span kind, name, inputs, and outputs are captured for you. Use them when you want to trace whole functions rather than specific code blocks.
### Enable the Decorators
Python needs a one-line wrap step to surface the decorator methods; JS/TS works with any OpenTelemetry tracer.
The tracer returned by `arize.otel.register()` is a standard OpenTelemetry tracer and does **not** expose the decorator methods directly. Wrap it in an `OITracer` from `openinference-instrumentation` to enable `chain`, `agent`, `tool`, and `llm`:
```bash theme={null}
pip install openinference-instrumentation
```
```python theme={null}
from arize.otel import register
from openinference.instrumentation import OITracer, TraceConfig
tracer_provider = register(
space_id="YOUR_SPACE_ID",
api_key="YOUR_API_KEY",
project_name="YOUR_PROJECT_NAME",
)
tracer = OITracer(tracer_provider.get_tracer(__name__), config=TraceConfig())
```
`OITracer` is a thin wrapper from `openinference-instrumentation` that adds the `chain`, `agent`, `tool`, and `llm` decorator methods on top of a standard OpenTelemetry tracer. Without it, calling `@tracer.chain` raises `AttributeError`.
Use the OpenTelemetry tracer set up above, then install `@arizeai/openinference-core`:
```bash theme={null}
npm install --save @arizeai/openinference-core @opentelemetry/api
```
The wrapper functions (`traceChain`, `traceAgent`, `traceTool`, `withSpan`) accept any OpenTelemetry tracer — no wrapping required.
```typescript theme={null}
import { trace } from "@opentelemetry/api";
import {
traceChain,
traceAgent,
traceTool,
withSpan,
} from "@arizeai/openinference-core";
const tracer = trace.getTracer(SERVICE_NAME);
```
With the tracer ready, decorate functions for each OpenInference span kind.
### Trace a Chain Step
Use `@tracer.chain` for a generic step in a pipeline — pre-processing, post-processing, routing logic, or any function that isn't specifically an agent, tool, or LLM call.
```python theme={null}
@tracer.chain
def preprocess(input: str) -> str:
return input.strip().lower()
```
Override the span name with `name=`:
```python theme={null}
@tracer.chain(name="custom-chain-name")
def preprocess(input: str) -> dict:
return {"normalized": input.strip().lower()}
```
The decorator captures the function arguments as the span input, the return value as the span output, and tags the span as `CHAIN`.
```typescript theme={null}
const preprocess = traceChain(
(input: string): string => {
return input.trim().toLowerCase();
},
{ name: "preprocess" }
);
preprocess("Hello, World");
```
### Trace an Agent
Use `@tracer.agent` for the top-level function of an agentic workflow. The decorator behaves identically to `@tracer.chain` but tags the span as `AGENT`.
```python theme={null}
@tracer.agent
def run_agent(input: str) -> str:
plan = preprocess(input)
return execute_plan(plan)
```
With a custom name:
```python theme={null}
@tracer.agent(name="research-agent")
def run_agent(input: str) -> str:
...
```
```typescript theme={null}
const runAgent = traceAgent(
async (input: string): Promise => {
const plan = await preprocess(input);
return executePlan(plan);
},
{ name: "research-agent" }
);
```
For richer agent visualizations, combine `@tracer.agent` with graph node attributes — see [Agent trajectory](/docs/ax/instrument/agent-trajectory).
### Trace a Tool Call
Use `@tracer.tool` to instrument a function that the agent calls as a tool. By default the decorator reads the function name, docstring, and signature to populate the tool name, description, and parameters on the span.
```python theme={null}
@tracer.tool
def get_weather(city: str, units: str = "fahrenheit") -> str:
"""Look up the current weather for a city."""
return fetch_weather(city, units)
```
Override any of the inferred fields explicitly:
```python theme={null}
@tracer.tool(
name="weather-lookup",
description="Returns the current temperature and conditions for a city.",
parameters={"city": "string", "units": "string"},
)
def get_weather(city: str, units: str = "fahrenheit") -> str:
return fetch_weather(city, units)
```
```typescript theme={null}
const getWeather = traceTool(
async (city: string, units: string = "fahrenheit"): Promise => {
return fetchWeather(city, units);
},
{ name: "get-weather" }
);
```
### Trace an LLM Call
Use `@tracer.llm` to instrument a function that calls a language model. Unlike the other decorators, it accepts optional `process_input` and `process_output` callbacks that map the function's arguments and return value into OpenInference LLM attributes (messages, model name, token counts).
Basic usage — the entire `messages` list is recorded as input and the returned string as output:
```python theme={null}
from openai import OpenAI
from openai.types.chat import ChatCompletionMessageParam
client = OpenAI()
@tracer.llm
def invoke_llm(messages: list[ChatCompletionMessageParam]) -> str:
response = client.chat.completions.create(
model="gpt-5.5",
messages=messages,
)
return response.choices[0].message.content or ""
```
For richer attribute capture (token counts, model name, individual messages), supply `process_input` and `process_output` callbacks that return OpenInference attribute dicts:
```python theme={null}
from openinference.instrumentation import (
TokenCount,
get_llm_input_message_attributes,
get_llm_model_name_attributes,
get_llm_output_message_attributes,
get_llm_token_count_attributes,
)
def process_input(messages, model, temperature=None):
return {
**get_llm_input_message_attributes(messages),
**get_llm_model_name_attributes(model),
}
def process_output(response):
msg = response.choices[0].message
return {
**get_llm_output_message_attributes(
[{"role": msg.role, "content": msg.content}]
),
**get_llm_token_count_attributes(
TokenCount(
prompt=response.usage.prompt_tokens,
completion=response.usage.completion_tokens,
total=response.usage.total_tokens,
)
),
}
@tracer.llm(process_input=process_input, process_output=process_output)
def invoke_llm(messages, model, temperature=None):
return client.chat.completions.create(
messages=messages,
model=model,
temperature=temperature,
)
```
`process_input`'s signature must match the decorated function's signature exactly — the decorator forwards the same arguments to both.
`withSpan` is the general-purpose wrapper. Pass `kind: "LLM"` and optional `processInput` / `processOutput` callbacks:
```typescript theme={null}
import OpenAI from "openai";
import { withSpan } from "@arizeai/openinference-core";
const openai = new OpenAI();
const invokeLLM = withSpan(
async (messages: Array<{ role: string; content: string }>): Promise => {
const response = await openai.chat.completions.create({
model: "gpt-5.5",
messages,
});
return response.choices[0].message.content || "";
},
{ name: "invoke-llm", kind: "LLM" }
);
```
With input/output processing:
```typescript theme={null}
const invokeLLM = withSpan(
async (messages, model, temperature) => {
return openai.chat.completions.create({ messages, model, temperature });
},
{
name: "invoke-llm",
kind: "LLM",
processInput: (messages, model, temperature) =>
processInput(messages, model, temperature),
processOutput: (response) => processOutput(response),
}
);
```
### Async and Streaming
The Python decorators detect coroutine functions automatically — no separate API. The `@tracer.llm` decorator additionally handles sync and async generators, so streaming completions are traced end-to-end and the captured output is the full concatenated stream.
```python theme={null}
from typing import AsyncGenerator
from openai import AsyncOpenAI
async_client = AsyncOpenAI()
@tracer.llm
async def stream_llm(messages) -> AsyncGenerator[str, None]:
stream = await async_client.chat.completions.create(
model="gpt-5.5",
messages=messages,
stream=True,
)
async for chunk in stream:
if chunk.choices[0].delta.content:
yield chunk.choices[0].delta.content
```
`withSpan` and the specialized wrappers (`traceChain`, `traceAgent`, `traceTool`) are async by default — wrap an `async` function and the span is closed when the returned promise resolves.
### Patch Methods You Don't Own
Apply the decorators imperatively to trace third-party SDK methods without subclassing them.
```python theme={null}
wrap = tracer.llm
client.chat.completions.create = wrap(client.chat.completions.create)
```
After this assignment every call to `client.chat.completions.create(...)` produces an LLM span automatically.
```typescript theme={null}
client.chat.completions.create = withSpan(
client.chat.completions.create.bind(client.chat.completions),
{ name: "openai.chat.completions.create", kind: "LLM" }
);
```
### Combine with Session and User Context
The span-kind decorators stack with the context decorators from [Set up sessions](/docs/ax/instrument/set-up-sessions) and the attribute helpers from [Customize your traces](/docs/ax/instrument/customize-your-traces).
```python theme={null}
from openinference.instrumentation import using_attributes
@using_attributes(session_id="abc-123", user_id="user-42")
@tracer.agent
def run_agent(input: str) -> str:
return generate_response(input)
```
The resulting `AGENT` span carries `session.id`, `user.id`, the function input, and the function output.
Use `setSession`, `setUser`, or `setAttributes` from `@arizeai/openinference-core` to push context, then call the wrapped function inside `context.with`:
```typescript theme={null}
import { context } from "@opentelemetry/api";
import { setSession, setUser } from "@arizeai/openinference-core";
const runAgent = traceAgent(
async (input: string) => generateResponse(input),
{ name: "run-agent" }
);
context.with(
setUser(setSession(context.active(), { sessionId: "abc-123" }), { userId: "user-42" }),
() => runAgent("hello")
);
```
### Decorate TypeScript Class Methods
TypeScript 5 introduced standard decorators, and `@arizeai/openinference-core` ships an `@observe` decorator for tracing class methods while preserving `this`.
```typescript theme={null}
import { observe } from "@arizeai/openinference-core";
class Agent {
@observe({ kind: "AGENT", name: "Agent.run" })
async run(input: string): Promise {
return this.generate(input);
}
@observe({ kind: "CHAIN" })
async generate(input: string): Promise {
return `processed: ${input}`;
}
}
```
# Learn More
* **Group traces into conversations** — attach `session.id` and `user.id` to your manual spans to follow multi-turn interactions. See [Set up sessions](/docs/ax/instrument/set-up-sessions).
* **Mix auto + manual** — let auto-instrumentors handle LLM calls while you keep manual CHAIN and TOOL spans for custom logic. See [Combine auto + manual](/docs/ax/instrument/combining-auto-and-manual).
* **Visualize agent execution** — set `graph.node.id` and `graph.node.parent_id` to render agent graphs in the UI. See [Agent trajectory](/docs/ax/instrument/agent-trajectory).
* **Track costs** — turn token counts on your LLM spans into per-span and per-trace cost. See [Track costs](/docs/ax/instrument/track-costs).
* **Configure for production** — switch to `BatchSpanProcessor`, add resource attributes, route to multiple projects. See [Configure your tracer](/docs/ax/instrument/configure-your-tracer).
* **Mask sensitive data** — hide PII in inputs/outputs before spans leave your app. See [Mask and redact data](/docs/ax/instrument/mask-and-redact-data).
* **Scale with OTEL Collector** — centralize routing, handle async context, sample at volume. See [Advanced patterns](/docs/ax/instrument/advanced-patterns).
# FAQs
**Q: Do I *have* to use an SDK that supports OpenInference?**
**A:** No; you can use any OpenTelemetry-compatible tracer. But if you instrument using the OpenInference schema (span kinds + attributes) you'll get better integration (analytics, visualization) in Arize AX.
**Q: What if I'm capturing sensitive data (PII) in spans or attributes?**
**A:** When using manual instrumentation, you must handle masking, redaction, or encryption as appropriate. See [Mask and redact data](/docs/ax/instrument/mask-and-redact-data).
**Q: Why does `@tracer.chain` raise `AttributeError` on a tracer returned by `arize.otel.register()`?**
**A:** `register()` returns a standard OpenTelemetry tracer provider whose tracers don't carry the OpenInference decorator methods. Wrap the tracer in `OITracer` as shown in [Enable the decorators](#enable-the-decorators).
***
## Next step
Customize your spans with attributes, events, prompt templates, and more: