What Is A Trace?

A trace is the record of one request or run through a system. It is made of spans, where each span is one unit of work with a start time, an end time, a set of attributes, and a parent. Read in order, the spans in a trace reconstruct what the system actually did, how long each part took, and which step caused which.

That vocabulary comes from OpenTelemetry and applies to an LLM application exactly as it applies to a web service. What changes is what counts as a unit of work. Instead of a request handler, a database query, and a cache read, the spans are a model call, a retrieval step, a tool invocation, and the agent step that decided to make them.

Traces matter more for AI systems than for ordinary services because you cannot read the execution path off the source code. The decisions that matter happen at runtime: which tool to call, what to retrieve, whether to retry, how to plan the next step. Source code tells you what the system could do. The trace tells you what it did.

Key takeaways

• A trace is one request or run. A span is one unit of work inside it. Parent pointers link spans into a tree, which is what lets you read a run in order and in nesting.
• A session is a set of traces belonging to the same ongoing interaction, so the hierarchy runs span, then trace, then session.
• For an LLM application the spans worth having are the model call, the retrieval step, each tool call, and the agent step that wrapped them, with inputs and outputs as attributes.
• A span with status OK is not evidence of success. Most AI failures are wrong content inside a span that completed normally, which is why span attributes matter as much as span timing.
• Traces are the input to evaluation. A score with no span to attach to cannot tell you which step to change.

Spans, and what one span holds

A span is one operation with a duration. Its standard fields:

• Name. What the operation was, such as retrieve_documents or openai.chat.
• Start and end timestamps. Duration, and a position on a timeline.
• Trace ID. Shared by every span in the same trace.
• Span ID and parent span ID. The parent pointer is the whole structure. A span with no parent is the root span, usually the request or the agent run.
• Status. OK, ERROR, or unset, plus an exception record when something threw.
• Attributes. Typed key-value pairs describing this operation: model name, token counts, the prompt, the completion, tool arguments, retrieved document IDs, the user or tenant.
• Events. Timestamped notes inside the span, useful for a first-token time or a retry.

For LLM work the attributes matter more than they do on a database span, because the content is what you are debugging. Naming them consistently is what OpenInference semantic conventions exist for, and it is why the span, trace, and session model is worth adopting as written rather than inventing per service.

Span, trace, and session

These three words carry the rest of the glossary, so state them precisely.

• A span is one unit of work, with a name, a start, an end, a status, attributes, and a pointer to its parent span.
• A trace is the record of one request or run through a system, made of the spans linked by those parent pointers into a single tree.
• A session is a group of traces that belong to the same ongoing interaction, such as a multi-turn conversation with one user or one long-running task.

Span nests inside trace, trace groups into session. One user turn in a chat is usually one trace, and the full conversation is the session. Session is a convention rather than an OpenTelemetry primitive: it is an identifier stamped onto every span so traces can be grouped later. Assigning that ID and returning it with the response is what makes a user complaint findable.

What a trace looks like for an LLM application

A single question to a RAG chatbot, traced, is roughly this shape:

• chat_request (root span, 3.1s)
• embed_query (40ms) with the embedding model and the input text
• vector_search (85ms) with top_k, the filter, and the returned chunk IDs and scores
• llm.chat (2.9s) with the assembled messages, the model, the completion, and token counts

An agent run is the same idea with more depth and a loop:

• agent_run (root span, 22s)
• llm.chat step 1, which returns a tool call
• tool.search_orders with the arguments the model emitted and the raw result
• llm.chat step 2, reading that result
• tool.issue_refund, which errors with a 429
• tool.issue_refund retry, which succeeds
• llm.chat step 3, the final answer

Read that tree and the run’s trajectory is visible: how many model calls it took, which tool it reached for, what arguments it passed, where the time and tokens went. That is the level at which agent failures are diagnosable, and it is why tracing and evaluation are treated as one workflow rather than two.

What traces are used for

Debugging a specific failure. Someone reports a wrong answer. You open that trace and read forward from the root, not backward from the error, because the span that surfaced the problem is usually downstream of the span that caused it.

Evaluation. Evals score things found in spans: retrieved chunks against the question, the answer against those chunks, tool arguments against the schema, the final output against the task. Writing the score back onto the span keeps it addressable, so a bad number leads to a step rather than to a dashboard.

Aggregates and a durable record. Once every run is a trace, tokens per run and tool error rate per tool are queries rather than guesses, and the accumulated traces describe how the system behaves under real traffic. That is the raw material for regression datasets and for the context graphs built out of agent traces. Logs get rotated away. Traces are worth keeping longer than you think.

FAQ

What is the difference between a trace and a span?

A span is one operation. A trace is all the spans for one request or run, connected by parent pointers. If you are looking at a single model call, that is a span. If you are looking at the model call plus the retrieval that fed it plus the tool call that followed, that is a trace.

What are traces in AI systems specifically?

Structurally the same as in any distributed system, with a different payload. An AI trace carries content, not just timings: the prompt actually sent, the documents actually retrieved, the arguments actually passed to a tool, and the text the model returned. Latency still matters, and the reason to keep the trace is that the content is where wrong answers come from.

How is a trace different from a log?

A log line is an independent event with a timestamp. A trace is a structured tree with timing and causality, so you can tell that this retrieval fed that model call. You can reconstruct some of a trace from well-tagged logs, and it is harder than instrumenting spans in the first place.

How much should a span record?

Enough to reproduce the decision without rerunning the system: the full input, the full output, the parameters, and the identifiers. Prompts and completions are large, so this is a real storage cost rather than a free one. Truncating the prompt is the choice people regret most, because it hides the most common failure, which is a context problem.