Sample code: Arize AX and Dynatrace dual export
Collector config, Python and TypeScript examples, and a script that checks both backends.
- Arize AX gets the AI spans (agents, LLM calls, tools, retrievers) with full prompts and responses, for evaluation and debugging. Every trace has a root, so it appears in the Traces view.
- Dynatrace gets every span (HTTP, database, internal, and the AI spans without their prompt content), the logs, and the metrics.
Choose an approach
You can split the telemetry in two places:- Through an OpenTelemetry Collector. The app exports to one OTLP endpoint, and the collector filters, redacts, and fans out to both backends.
- In application code. One
TracerProviderwith two span processors, one for each backend. No extra infrastructure, which suits serverless and edge runtimes.
Before you start
You need:- An Arize AX account, with your API key and Space ID
- A Dynatrace environment and an ingest token. A platform token (
dt0s16.*) needsopenpipeline:traces:ingest,openpipeline:logs:ingest, andopenpipeline:metrics:ingest. An access token (dt0c01.*) needsopenTelemetryTrace.ingest,logs.ingest, andmetrics.ingest. - Python 3.10 or later, Node 20 or later for the TypeScript example, and an OpenAI API key for the example agent
- Optional, to check both backends from the command line: the AX CLI and a Dynatrace platform token with
storage:logs:read,storage:metrics:read,storage:spans:read, andstorage:buckets:read
Set up the sample
The sample app is a small FastAPI support agent. Each request runs a sqlite query, an OpenAI tool-calling loop, a tool that queries the database, and a tool that calls a weather API over HTTP. It emits logs and three metrics.1
Clone the sample
2
Create the .env file
ARIZE_SPACE_ID, ARIZE_API_KEY, DYNATRACE_ENV_URL (for example https://{your-environment-id}.live.dynatrace.com, without a path), DYNATRACE_API_TOKEN, and OPENAI_API_KEY. Set DYNATRACE_AUTH_SCHEME to Bearer for platform tokens or Api-Token for access tokens.3
Install the Python dependencies
Send through an OpenTelemetry Collector
The app needs no Arize or Dynatrace credentials and no filtering code. It exports traces, logs, and metrics with the standardOTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318. The collector config is collector/config.yaml. It works with the Dynatrace distribution of the collector (0.60.0) and with otelcol-contrib (0.165.0).
How the config fans out
Shared processing, such as sampling, runs once in
traces/in, before the fan-out, so both backends keep the same traces. The forward connector copies every span into each branch. It ships in both distributions, and unlike the routing connector it sends each span to every branch, so AI spans reach both backends.
The Arize branch keeps spans that follow the OpenInference or the OpenTelemetry GenAI semantic conventions. Arize AX ingests gen_ai.* spans natively and derives the span kind from gen_ai.operation.name.
otlp_http exporter, set compression: none, because Arize’s HTTP endpoint rejects gzip bodies.
cumulative_to_delta. Logs pass through unchanged: the SDK puts the trace and span id on each record, and Dynatrace exposes them as trace_id and span_id.
Run it
1
Start the collector
bin/, validates the config, and runs it. Use DISTRO=contrib scripts/run-collector.sh for otelcol-contrib, or docker compose up -d to run the collector in a container.2
Send a request
In another shell:The app prints the trace id:
docs/collector.md, covers tail sampling, scaling with the load_balancing exporter, Kubernetes, and the debug overlay.
Send from application code
Use oneTracerProvider with two span processors. Each processor sees every span that ends and decides what to forward:
- The Arize processor keeps only AI spans and makes the top AI span the root.
- The Dynatrace processor exports every span through an exporter that removes AI content.
- A
LoggerProviderand aMeterProvider(with delta temporality) export to Dynatrace.
- Python
- TypeScript
The Python OpenInference packages don’t include a span filter or root processor, so the sample has a small one in Then add your HTTP and database instrumentation (
example/openinference_processors.py. OpenInferenceSpanProcessor forwards AI spans to the Arize exporter, and ContentRedactingExporter removes prompt and response attributes before the Dynatrace export.FastAPIInstrumentor, SQLite3Instrumentor, HTTPXClientInstrumentor, and so on) on the same provider.If you use arize.otel.register(), keep it as the only provider. Its own Arize exporter receives every span, including APM spans, so to keep APM spans out of Arize, build the provider as shown above.Run the sample:- Use one provider per process. Two
TracerProviders, for example one fromarize.otel.register()and one for Dynatrace, produce two separate traces for the same request. - Don’t set
trace_idas a log attribute yourself. The SDK already sends it as a top-level field. If you add it again, Dynatrace storestrace_idas an array andfilter trace_id == "..."stops matching.
traceparent header. Each service exports with its own two processors, and both backends join the spans into one trace by id.
The in-code guide in the sample, docs/in-code-instrumentation.md, covers sampling, flushing in serverless functions, and the tradeoffs of sending AI spans to Dynatrace.
How traces stay rooted in Arize
Arize lists traces by their root span, the span with no parent. The top AI span of a request usually has an APM parent, such as the HTTP server span, and the Arize filter drops that parent. Without a fix, Arize stores the AI spans but has no root for the trace, so the trace is missing from the Traces view. Both setups fix this by default, and neither changes what Dynatrace receives.- Collector
- In code
The In Arize, the request span is the root and the AI spans sit under it at their real positions. No parent ids change, so nested agents stay nested. The AI spans themselves never pass through
traces/arize-root branch sends the trace’s root span to Arize as a CHAIN:tail_sampling/traces-with-ai-spanswaitsdecision_wait(30 seconds by default) and passes a trace on only if it contains an AI span. Requests without AI spans send nothing to Arize.filter/non-ai-trace-rootkeeps only the span with no parent, and only if it is not already an AI span.transform/root-as-chainsetsopeninference.span.kindtoCHAINon that span.
tail_sampling.This works when the top AI span’s parent is the trace root and all spans of a trace reach the same collector instance. With several collector replicas, route spans by trace id with the load_balancing exporter. If you can change the app, tagging the request span as a CHAIN there is still better, because it also puts the request input and output on the root.If you build your own collector, Arize’s filter-remap-processor is an alternative. It buffers each trace, drops the non-AI spans, and points every kept span at its nearest kept ancestor. It is not part of the Dynatrace distribution or
otelcol-contrib, so it needs a custom build with the OpenTelemetry Collector Builder.Verify both backends
Look up the trace id the app printed in each backend.1
Find the trace in Arize AX
Open the project named by
ARIZE_PROJECT_NAME and find the trace id in the Traces view. The trace has one root: POST /ask as a CHAIN through the collector, or support_agent in code. With the AX CLI:2
Find the trace and logs in Dynatrace
Open the trace id in the Distributed Tracing app, or run these queries in a notebook:Dynatrace has the full tree: the HTTP, database, and AI spans, with one root (
POST /ask) and no input.value or output.value on the AI spans.3
Run the automated check
scripts/verify.py sends one request per scenario and checks what each backend stored: all AI spans and only AI spans in Arize, one root with no dangling parents, the trace listed in Traces, the full redacted trace in Dynatrace, correlated logs, and metrics. It needs the AX CLI and DYNATRACE_QUERY_TOKEN in .env.TESTING.md lists every scenario and check.Troubleshooting
Learn more
OpenTelemetry Collector
Collector concepts, deployment models, and common pitfalls.
Advanced patterns
Collector authentication and dynamic project routing for Arize AX.
Semantic conventions
How Arize AX maps OpenInference and OTel GenAI attributes.
Dynatrace OpenTelemetry Collector
Install and configure the Dynatrace distribution of the collector.