Model monitoring for LLM and agent systems

Learn how model monitoring works for LLMs and agents: the metrics to track, how evals and monitors fit together, and how to set alert thresholds.

Model monitoring for an LLM or agent system means tracking whether outputs stay good and whether the steps behind them stay sound. It relies on evaluation scores and trace data alongside the latency and cost numbers, in place of the accuracy and drift metrics that classical ML monitoring depends on.

Classical monitoring grew up around models with fixed input features and a label that arrives later. An LLM or agent takes free text and produces free text, often making a chain of decisions in between, so a response can come back quickly with no errors and still be wrong. This guide covers what to monitor, what agents add to the list, and how to set up alerts that fire on real regressions.

How is monitoring an LLM different from monitoring a classical ML model?

Classical model monitoring splits into monitor types that each answer a different question:

  • Performance monitors use ground truth to track metrics like AUC and MAPE.
  • Drift monitors compare production data against a baseline using PSI, KL divergence, JS distance, or the KS statistic.
  • Data quality monitors check inputs for problems like empty values and new categories.

An LLM application changes what each of those looks like:

Metric / Dimension Classical ML LLM and Agent Systems
Quality signal Ground truth labels, often delayed Evaluation scores from LLM judges and code checks, calibrated against human annotations
Input changes Feature and prediction drift Shifts in what users ask and what the system retrieves
Data problems Percent empty, cardinality, new values Stale or missing context and malformed tool results
Unit of observation One prediction A span, a full trace, or a multi-turn session

What should you monitor in an LLM application?

Start with the metrics below, which apply to any application that calls a model:

  • Output quality: Eval scores for the failure types that matter in your app, such as hallucination or groundedness. A label like “ungrounded” becomes a rate you can chart over time and alert on.
  • Latency: P95 latency alongside the average, since slow outliers disappear inside a mean.
  • Errors: Failed calls and error status, counted per trace when one failure produces many error spans.
  • Token usage: Prompt and completion tokens, plus the total, over time. A sudden jump often points to a change in a prompt template or in how much context gets retrieved.
  • Cost: Spend by model and provider, which follows from token counts and each model’s per-token price.
Try Arize AX

Build better agents with Arize

Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.

Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.

What does monitoring an agent add?

An agent adds a path to the output. The final response can look fine while the run behind it looped, called the wrong tool, or never changed the system it claimed to update. Tracking agent observability signals on top of the LLM basics catches those runs:

  • Task completion: The metric that shows the agent did its job, confirmed in the system where the action happened. A run can report success while the target system never changed.
  • Tool errors and retries: Tool failures and retries, plus handoff errors, with the arguments, results, and exceptions captured so you can see what went wrong.
  • Loops and repeated calls: Repeated calls that return the same evidence, routing cycles, and handoffs that lose context. Raw step count alone says little about quality, since a longer path can reflect necessary recovery.
  • Path efficiency: Agent convergence divides the optimal path length by the actual path length for similar queries. A score near 1 means runs take close to the optimal path.
  • Session-level outcomes: Session-level evaluations judge a whole multi-turn conversation, including whether the user reached their goal, which span-level checks can miss.
  • Cost per successful task: Token spend divided by completed tasks, tracked next to P95 latency so a cost change can be explained by an outcome change.

How do evals and monitors work together?

Evaluations score spans, traces, or sessions. Monitors watch a metric over time and alert when it crosses a threshold. The two connect through online evals, which run an evaluator continuously on production traces and attach the score to each one, so an eval label becomes a metric a monitor can watch.

Running an evaluator on every request gets expensive at volume, so sample. The Arize AX guide to online evals scales the sampling rate with traffic:

  • Low-volume or critical apps: 100%
  • High-volume apps: 10% to 50%
  • Very high volume: 1% to 5%

Start at 10% to 20%, and raise the rate once you’ve confirmed the evaluator works correctly. Review eval results against human annotations before you trust them enough to alert on. TheFork’s online evals setup shows what this looks like in production.

How do you set alert thresholds for LLM metrics?

Pick the threshold type by how well you know the normal range:

  • Static thresholds fit hard limits, such as a maximum cost per trace or a minimum eval pass rate.
  • Automatic thresholds learn the normal range from historical data, which suits metrics that vary with traffic, such as latency and token usage. Raising the sensitivity sends more alerts.

Set upper and lower bounds when both directions matter. A drop in output tokens can signal empty responses just as a spike signals runaway generation.

Alert on rates rather than single events, such as the share of responses labeled ungrounded in a window. LLM outputs vary run to run, so one bad response rarely deserves a page, while a sustained shift in the rate is worth investigating.

How do you start monitoring an LLM or agent system?

Start small and build up in layers, so each step gives you a signal you can act on before you add the next one:

  1. Pick one outcome metric. Choose the number that shows the system did its job, such as task completion for an agent or groundedness for a retrieval app.
  2. Chart the basics. Put P95 latency, error rate, token usage, and cost on a dashboard for the project.
  3. Add an eval on sampled traffic. Start at 10% to 20% sampling and score the failure type you care about most.
  4. Set monitors on the metrics you chart. Add a notification destination so regressions page you instead of waiting to be noticed.
  5. Review flagged traces weekly. Turn each recurring failure into a new eval so the monitor set grows with what you learn.

Model monitoring for LLMs and agents with Arize AX

Arize AX covers each layer of this setup in one place, from the metrics you chart to the monitors that alert on them:

  • Tracing project monitors: Create monitors on any span attribute, such as latency or an eval label like eval.groundedness.label, or on a custom metric like hallucination rate. Thresholds can be automatic or static, with optional upper and lower bounds, and alerts go to email, Slack, PagerDuty, OpsGenie, Microsoft Teams (through a webhook), or webhooks.
  • Online evals: Run any evaluator continuously, including LLM judges and code evaluators, at the span, trace, or session level, with sampling rates and filters. Span scores land on the span they evaluate, and trace and session scores land on root spans.
  • Dashboards and cost tracking: The Tracing Project Overview template charts trace volume, errors, latency, token usage, cost, and eval scores, and cost tracking computes spend from token counts and per-model rates.
  • Signal: Signal scans a tracing project’s traces on a schedule, every 6 hours by default, and groups recurring failures into issues with trace evidence and a proposed fix. It’s on every AX plan, with 10 issues a month on Free, 25 on Pro, and unlimited on Enterprise. Opening fix pull requests against your repo is Enterprise-only.

Start here

FAQs: Model monitoring for LLMs and agents

What’s the difference between model monitoring and model observability?

Monitoring watches metrics over time and alerts when one crosses a threshold. Observability keeps the traces, spans, and sessions that explain why it did. Monitors tell you something changed, and the trace data lets you find the cause.

Can you monitor an LLM without ground truth labels?

Yes. Evals stand in for labels: LLM judges handle judgment calls such as groundedness or relevance, and code checks handle deterministic rules. Calibrate both against a sample of human annotations so you know the scores can be trusted.

How often should you run evals on production traffic?

Run them continuously on a sample. Low-volume or critical applications can score every request, while high-volume applications usually score 10% to 50% of traffic and very high-volume ones 1% to 5%. Sampling applies at the highest evaluator level, so a session-level eval samples sessions.

Do drift monitors still matter for LLM applications?

Drift monitors compare production data against a baseline and still apply to ML models with structured features. For an LLM application, the more direct signals are eval scores and a view of what users are asking, because a shift in either tends to show up in quality metrics.

How do you monitor LLM cost?

Capture token counts on every LLM span and price them by model and provider. Chart cost over time, set an alert on cost per trace, and track cost per successful task for agents so a cost increase can be tied to an outcome.

Get the latest on AI & Observability

Sign up for our newsletter, The Evaluator—and stay in the know with updates and new resources:

Don’t ship vibes.

Arize gives AI teams observability and evals to understand and improve agent performance.