> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Tracing Concepts
> How Arize AX tracing works — OpenTelemetry, OpenInference, traces, spans, and the instrumentation pipeline
Tracing is how you see inside your LLM app. Without it, debugging is guesswork. With it, every call, every tool invocation, every retrieval is visible.
# How Tracing Works
Your LLM application handles requests — each one might call a model, retrieve documents, run tools, and return a response. A **trace** captures that entire journey as a tree of **spans**. Each span represents one operation (an LLM call, a retrieval, a tool invocation) with its input, output, timing, and metadata.
Under the hood, this is powered by two open standards:
* **[OpenTelemetry](https://opentelemetry.io)** (OTel) — the universal, vendor-agnostic framework for collecting and exporting telemetry data. Your instrumentation isn't locked to Arize AX — it works with any OTel-compatible backend.
* **[OpenInference](https://github.com/Arize-ai/openinference)** — GenAI-specific semantic conventions built on top of OTel, created by Arize. OpenInference defines the attributes that make traces meaningful for LLM applications: span kinds, message formats, token counts, cost, and more. Arize accepts standard OTel spans and reads OpenInference attributes for richer GenAI visualization.
Together, they give you [30+ native integrations](/ax/integrations), portability across backends, and a standardized format for your trace data. Here's how it all connects:
1. **Instrumentation** wraps your function calls (automatically via [integrations](/ax/integrations), or manually) and captures span data following OpenInference semantic conventions.
2. **An exporter** sends the spans to Arize AX via [OTLP](https://opentelemetry.io/docs/specs/otlp/) (gRPC by default).
3. **The Arize collector** ingests and visualizes them so you can explore, filter, and debug.
# What's Inside a Trace
Now that you know how data flows from your app to Arize AX, let's look at what a trace contains.
Each span in the tree is assigned a **span kind** — the type of operation it represents:
| Span Kind | Description |
| :------------ | :----------------------------------------------------- |
| **LLM** | Call to an LLM for a completion or chat |
| **Tool** | API or function invoked on behalf of an LLM |
| **Agent** | Root span containing a set of LLM and tool invocations |
| **Retriever** | Data retrieval query for context from a datastore |
| **Chain** | The starting point and link between application steps |
| **Embedding** | Encoding of unstructured data |
| **Guardrail** | Validates LLM inputs/outputs for safety and compliance |
| **Reranker** | Relevance-based re-ordering of documents |
| **Evaluator** | Evaluation process, type, and results |
| **Audio** | Audio or voice processing operations |
If you don't set a span kind, it defaults to **Unknown**. Here's what this looks like in practice — an agent trace with Agent, Retriever, Tool, and LLM spans forming a tree:
## Span Attributes
Each span also carries **attributes** — key-value pairs like `llm.model_name`, `llm.input_messages`, `input.value`, `output.value`. These are the [OpenInference semantic conventions](https://github.com/Arize-ai/openinference) — auto-instrumentation sets them for you, and you can add your own via [Customize your traces](/ax/instrument/customize-your-traces). Check the OpenInference source for every attribute defined per span kind:
***
## Next step
You know how tracing works and what's inside a trace. Now get traces flowing from your app:
# What's Inside a Trace
Now that you know how data flows from your app to Arize AX, let's look at what a trace contains.
Each span in the tree is assigned a **span kind** — the type of operation it represents:
| Span Kind | Description |
| :------------ | :----------------------------------------------------- |
| **LLM** | Call to an LLM for a completion or chat |
| **Tool** | API or function invoked on behalf of an LLM |
| **Agent** | Root span containing a set of LLM and tool invocations |
| **Retriever** | Data retrieval query for context from a datastore |
| **Chain** | The starting point and link between application steps |
| **Embedding** | Encoding of unstructured data |
| **Guardrail** | Validates LLM inputs/outputs for safety and compliance |
| **Reranker** | Relevance-based re-ordering of documents |
| **Evaluator** | Evaluation process, type, and results |
| **Audio** | Audio or voice processing operations |
If you don't set a span kind, it defaults to **Unknown**. Here's what this looks like in practice — an agent trace with Agent, Retriever, Tool, and LLM spans forming a tree:
## Span Attributes
Each span also carries **attributes** — key-value pairs like `llm.model_name`, `llm.input_messages`, `input.value`, `output.value`. These are the [OpenInference semantic conventions](https://github.com/Arize-ai/openinference) — auto-instrumentation sets them for you, and you can add your own via [Customize your traces](/ax/instrument/customize-your-traces). Check the OpenInference source for every attribute defined per span kind: