AI observability is the practice of instrumenting an AI system so that you can determine, from recorded data, why it produced the output it did. The recorded data is traces of what ran, the inputs and outputs at each step, evaluation scores attached to those steps, and human feedback where it exists.
It borrows the word from software observability, where the definition is the ability to understand a system’s internal state from its external outputs. The reason it needs a separate name is that the classic signals answer a question AI systems rarely fail at. A wrong answer returns 200. It is fast. It logs nothing unusual. Every dashboard is green and the user got a fabricated refund policy.
So the object of observation changes. In a conventional service you are watching for errors, saturation, and latency. In an AI system you are watching for the quality of content produced by a component whose behavior is not defined by your code, and doing it without a correct answer to compare against, on live traffic, at a volume nobody can read by hand.
Key takeaways
- AI observability answers why a specific output happened, using traces that record the actual inputs and outputs of every step, not just timings and status codes.
- Traditional metrics and logs miss the dominant failure mode, because a wrong answer is a successful request by every conventional measure.
- The signal set is traces plus evaluations plus human feedback. Evaluations supply the quality judgment that no infrastructure metric can, and they attach to spans so a bad score points at a step.
- The unit of investigation is a session or a run, not a request. Agent failures are trajectory failures, and the step that surfaced the problem is usually not the step that caused it.
- It does not produce ground truth. Judge-based scores are estimates with their own error rate, and they need calibration against human labels before a trend is worth acting on.
Why metrics and logs are not enough
The three classic signals still earn their place. Latency, token spend, and error rates matter, and an agent that times out or exhausts a rate limit is an ordinary operational problem with ordinary answers.
They stop being sufficient when the failure is semantic:
- The retriever returned three documents on the right topic, none of which answered the question, and the model answered anyway.
- The agent called the correct tool with a well-formed order ID that does not exist, got a clean
404, and reported that the order was cancelled. - The model dropped a constraint stated eleven turns earlier and gave advice that violates policy.
None of those raise an exception. To see them you need the content: the assembled input, the retrieved chunks, the tool arguments, the completion, and a judgment about whether the content was right. That is why AI observability is built on tracing with rich span attributes rather than on counters, and why OpenTelemetry ends up as the substrate even though the semantics being recorded are new.
The signals it collects
Traces and spans. One trace per request or run, with a span for each model call, retrieval, and tool invocation, carrying inputs, outputs, parameters, token counts, and status. The substrate everything else annotates.
Sessions. Traces grouped by conversation or task, so a multi-turn interaction can be read as one thing. Most reported problems arrive as a complaint about a conversation rather than a request.
Evaluations. Automated scores attached to spans: was the retrieved context relevant, was the answer grounded in it, were the tool arguments valid, did the run complete the task. Usually a judge model, sometimes deterministic code. These carry the quality judgment infrastructure signals cannot.
Human feedback. Thumbs, escalations, support tickets, and reviewer labels attached to the same spans. Small in volume, and what the automated scores get calibrated against.
How it is used
The loop is narrow and worth being explicit about, because tooling is often bought without it.
- Something surfaces: an alert on an eval score, a support ticket, a drop in completion rate.
- You find the specific sessions behind it, which requires a session ID returned with the response and searchable.
- You read a trace forward from the root span, looking for the first step whose output was wrong, not the last step where an error appeared.
- You locate the layer that owns the fix: retrieval, context assembly, a tool schema, an instruction, a control limit, or occasionally the model.
- The failing case becomes a row in an evaluation dataset so the same regression cannot ship twice.
Steps 3 and 4 consume time, and they are where instrumentation quality shows up. Doing this at volume, across many sessions rather than one at a time, is what an agent observability platform is meant to make routine.
What it does not give you
It does not tell you the right answer. There is usually no ground truth in production, so quality is estimated by judge models and sampled human review, both with measurable error rates.
It does not remove the cost decision. Storing full prompts and completions on every span is significant data volume and user content, which makes retention, redaction, and where that data lives a real design question. That is the practical reason deployment model matters as much as capability for teams with regulated data.
It does not replace evaluation before release. Observing production tells you what happened; offline evaluation on a fixed dataset is how you know whether a change is safe before it reaches anyone.
FAQ
How is AI observability different from AI monitoring?
Monitoring watches predefined signals and alerts when one crosses a threshold. Observability is the property that lets you investigate a question nobody predefined, which requires underlying data rich enough to answer questions you did not plan for. A run that returns a confidently wrong answer in 800 milliseconds with status 200 is indistinguishable from a good one at the infrastructure layer. You need both: monitors tell you something changed, and the trace data tells you why.
Is observability the same as evaluation?
They are separate and they need each other. Evaluation is scoring an output or a trajectory against criteria. Observability is the recorded execution data the scores attach to and that you read when a score is bad. Evaluation without tracing gives you a number with no path to a cause. Tracing without evaluation gives you a haystack.
What should I instrument first?
Trace the model calls, the retrieval step, and every tool call, with full inputs and outputs as span attributes, and emit a session ID returned to the client and logged with any feedback. That alone makes specific failures findable, which is the bottleneck for most teams. Add evaluations once you can read a trace, starting with the failure modes you already know you have.
Does it work without labeled data?
Yes, and that is the normal situation. Reference-free evaluations judge the output against the input rather than against a correct answer: whether the retrieved context is relevant to the question, whether the answer is supported by that context, whether the tool arguments match the request. Human labels are still needed, in small quantities, to check that those judgments agree with yours.
