> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Manual Instrumentation
> Create spans explicitly using the OpenTelemetry API for custom logic, unsupported frameworks, or fine-grained control
Auto-instrumentation covers supported frameworks. For everything else — custom logic, tool execution, unsupported frameworks, or fine-grained control — you create spans explicitly using the OpenTelemetry API with [OpenInference Semantic Conventions](https://github.com/Arize-ai/openinference).
# Set Up with Skills or Code
Three steps to add manual spans with your AI coding agent:
**Install skill**
```bash theme={null}
npx skills add Arize-ai/arize-skills --skill "arize-instrumentation" --yes
```
**Set up authentication**
```bash theme={null}
export ARIZE_API_KEY="YOUR_API_KEY"
export ARIZE_SPACE_ID="YOUR_SPACE_ID"
```
**Ask it to add manual spans**
```bash theme={null}
# Ask your AI coding agent:
"Wrap my custom tool functions with manual spans using OpenInference conventions"
```
Works with Cursor, Claude Code, Codex, and more. The skill picks the right span kinds, sets the right OpenInference attributes, and handles context propagation for you:
Set up the OpenTelemetry SDK, register a tracer provider with your Arize credentials, and create spans with [OpenInference](https://github.com/Arize-ai/openinference) attributes.
```bash theme={null}
pip install openinference-instrumentation opentelemetry-api opentelemetry-sdk arize-otel
```
```bash theme={null}
npm install @opentelemetry/api @opentelemetry/exporter-trace-otlp-proto @opentelemetry/resources @opentelemetry/sdk-trace-base @opentelemetry/sdk-trace-node @opentelemetry/semantic-conventions @arizeai/openinference-semantic-conventions openai
```
Requires Go 1.25+. Install [`arize-otel-go`](https://github.com/Arize-ai/arize-otel-go) for tracer setup, plus the OpenInference [`semantic-conventions`](https://github.com/Arize-ai/openinference/tree/main/go/openinference-semantic-conventions) and [`instrumentation`](https://github.com/Arize-ai/openinference/tree/main/go/openinference-instrumentation) packages — the first for typed attribute-key constants, the second for context-scoped helpers (`WithSession`, `WithUser`, `WithMetadata`, `WithTags`, `WithSuppression`):
```bash theme={null}
go get \
github.com/Arize-ai/arize-otel-go \
github.com/Arize-ai/openinference/go/openinference-instrumentation \
github.com/Arize-ai/openinference/go/openinference-semantic-conventions
```
Go to **Settings > API keys** to create your API key and to get the Space ID of your current space.
```python theme={null}
from arize.otel import register
tracer_provider = register(
space_id="YOUR_SPACE_ID",
api_key="YOUR_API_KEY",
project_name="YOUR_PROJECT_NAME",
)
tracer = tracer_provider.get_tracer(__name__)
```
You can also use environment variables `ARIZE_SPACE_ID`, `ARIZE_API_KEY`, and `ARIZE_PROJECT_NAME` instead of passing them directly.
```typescript theme={null}
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { SimpleSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
import { SEMRESATTRS_PROJECT_NAME } from "@arizeai/openinference-semantic-conventions";
const COLLECTOR_ENDPOINT = "https://otlp.arize.com";
const SERVICE_NAME = "manual-js-app";
const provider = new NodeTracerProvider({
resource: resourceFromAttributes({
[ATTR_SERVICE_NAME]: SERVICE_NAME,
[SEMRESATTRS_PROJECT_NAME]: SERVICE_NAME,
}),
spanProcessors: [
new SimpleSpanProcessor(
new OTLPTraceExporter({
url: `${COLLECTOR_ENDPOINT}/v1/traces`,
headers: {
'arize-space-id': 'your-arize-space-id',
'arize-api-key': 'your-arize-api-key',
},
})
),
],
});
provider.register();
```
`arizeotel.Register` returns a configured `*sdktrace.TracerProvider`, installs it as the global, sets the required `openinference.project.name` resource attribute, and reads `ARIZE_SPACE_ID` / `ARIZE_API_KEY` / `ARIZE_PROJECT_NAME` from the environment when the matching `Options` fields are unset.
```go theme={null}
package main
import (
"context"
"log"
"time"
arizeotel "github.com/Arize-ai/arize-otel-go"
"go.opentelemetry.io/otel"
)
func main() {
ctx := context.Background()
tp, err := arizeotel.Register(ctx, arizeotel.Options{
ProjectName: "manual-go-app",
})
if err != nil {
log.Printf("register tracer: %v", err)
return
}
defer func() {
// Never call log.Fatalf / os.Exit after a span starts — they
// skip this deferred Shutdown and in-flight spans never flush.
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_ = tp.Shutdown(shutdownCtx)
}()
tracer := otel.Tracer("manual-go-app")
_ = tracer
// ... your app ...
}
```
For EU-region spaces, pass `Endpoint: arizeotel.EndpointArizeEurope`. Set `ARIZE_SPACE_ID` and `ARIZE_API_KEY` as environment variables — never embed credentials in source.
Use your tracer to create spans — either as a context manager (to trace a specific block) or inline.
```python theme={null}
import openai
from opentelemetry.trace import Status, StatusCode
client = openai.OpenAI()
def run_agent(user_input: str) -> str:
with tracer.start_as_current_span("run-agent") as span:
span.set_attribute("openinference.span.kind", "CHAIN")
span.set_attribute("input.value", user_input)
response = call_llm(user_input)
span.set_attribute("output.value", response)
span.set_status(Status(StatusCode.OK))
return response
def call_llm(prompt: str) -> str:
with tracer.start_as_current_span("llm-completion") as span:
span.set_attribute("openinference.span.kind", "LLM")
span.set_attribute("input.value", prompt)
span.set_attribute("llm.model_name", "gpt-4o")
completion = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
)
result = completion.choices[0].message.content or ""
span.set_attribute("output.value", result)
span.set_status(Status(StatusCode.OK))
return result
```
```typescript theme={null}
import { trace, SpanStatusCode } from "@opentelemetry/api";
import { INPUT_VALUE, OUTPUT_VALUE, LLM_MODEL_NAME, SemanticConventions,
OpenInferenceSpanKind
} from "@arizeai/openinference-semantic-conventions";
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const tracer = trace.getTracer(SERVICE_NAME);
async function callLLM(prompt: string): Promise {
return tracer.startActiveSpan("call-llm", async (span) => {
span.setAttribute(SemanticConventions.OPENINFERENCE_SPAN_KIND, OpenInferenceSpanKind.LLM);
span.setAttribute(INPUT_VALUE, prompt);
span.setAttribute(LLM_MODEL_NAME, "gpt-4o");
const response = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: prompt }]
});
const result = response.choices[0].message.content || "";
span.setAttribute(OUTPUT_VALUE, result);
span.setStatus({ code: SpanStatusCode.OK });
span.end();
return result;
});
}
```
Use the [`openinference-semantic-conventions`](https://github.com/Arize-ai/openinference/tree/main/go/openinference-semantic-conventions) package for typed attribute-key constants — no raw strings. The example below shows the fully-manual pattern (CHAIN + LLM) for clients without a first-party instrumentor. If you're using `openai/openai-go` or `anthropics/anthropic-sdk-go`, drop the inner `llm-completion` span — the middleware emits it for you — and keep only the surrounding `CHAIN`.
```go theme={null}
package main
import (
"context"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
semconv "github.com/Arize-ai/openinference/go/openinference-semantic-conventions"
)
var tracer = otel.Tracer("manual-go-app")
func runAgent(ctx context.Context, userInput string) (string, error) {
ctx, span := tracer.Start(ctx, "run-agent")
defer span.End()
span.SetAttributes(
attribute.String(semconv.OpenInferenceSpanKind, semconv.SpanKindChain),
attribute.String(semconv.InputValue, userInput),
)
response, err := callLLM(ctx, userInput)
if err != nil {
span.SetStatus(codes.Error, err.Error())
return "", err
}
span.SetAttributes(attribute.String(semconv.OutputValue, response))
span.SetStatus(codes.Ok, "")
return response, nil
}
func callLLM(ctx context.Context, prompt string) (string, error) {
ctx, span := tracer.Start(ctx, "llm-completion")
defer span.End()
span.SetAttributes(
attribute.String(semconv.OpenInferenceSpanKind, semconv.SpanKindLLM),
attribute.String(semconv.InputValue, prompt),
attribute.String(semconv.LLMModelName, "gpt-4o"),
)
// ... call your LLM client, then set the output ...
result := "..."
span.SetAttributes(attribute.String(semconv.OutputValue, result))
span.SetStatus(codes.Ok, "")
return result, nil
}
```
Indexed attributes — e.g. message lists — are produced by helper functions: `semconv.LLMInputMessageRoleKey(0)`, `semconv.LLMInputMessageContentKey(0)`, and so on. See the [package reference](https://pkg.go.dev/github.com/Arize-ai/openinference/go/openinference-semantic-conventions) for the full set.
# Learn More
* **Group traces into conversations** — attach `session.id` and `user.id` to your manual spans to follow multi-turn interactions. See [Set up sessions](/ax/instrument/set-up-sessions).
* **Mix auto + manual** — let auto-instrumentors handle LLM calls while you keep manual CHAIN and TOOL spans for custom logic. See [Combine auto + manual](/ax/instrument/combining-auto-and-manual).
* **Visualize agent execution** — set `graph.node.id` and `graph.node.parent_id` to render agent graphs in the UI. See [Agent trajectory](/ax/instrument/agent-trajectory).
* **Track costs** — turn token counts on your LLM spans into per-span and per-trace cost. See [Track costs](/ax/instrument/track-costs).
* **Configure for production** — switch to `BatchSpanProcessor`, add resource attributes, route to multiple projects. See [Configure your tracer](/ax/instrument/configure-your-tracer).
* **Mask sensitive data** — hide PII in inputs/outputs before spans leave your app. See [Mask and redact data](/ax/instrument/mask-and-redact-data).
* **Scale with OTEL Collector** — centralize routing, handle async context, sample at volume. See [Advanced patterns](/ax/instrument/advanced-patterns).
# FAQs
**Q: Do I *have* to use an SDK that supports OpenInference?**
**A:** No; you can use any OpenTelemetry-compatible tracer. But if you instrument using the OpenInference schema (span kinds + attributes) you'll get better integration (analytics, visualization) in Arize AX.
**Q: What if I'm capturing sensitive data (PII) in spans or attributes?**
**A:** When using manual instrumentation, you must handle masking, redaction, or encryption as appropriate. See [Mask and redact data](/ax/instrument/mask-and-redact-data).
***
## Next step
Customize your spans with attributes, events, prompt templates, and more: