Data drift for an LLM system occurs when the inputs change. Users start asking about different things, phrase requests differently, arrive from a new locale or client, or the retrieved content and tool responses feeding the model stop being what they were last quarter. The model can be identical and the prompt untouched. The world around them moved.
That is the whole scope of the term, and it is worth being strict about, because several kinds of drift get discussed together and they call for different responses. A change in the system’s behavior is agent drift. A loss of validity in your measurement is evaluation drift. Accumulated edits to prompt text are prompt drift. Data drift is the input distribution moving, and it is the one that originates outside your system entirely.
It matters because everything downstream was fitted to the old distribution: the prompt written for the questions people used to ask, retrieval settings tuned for the queries you had, thresholds calibrated on last quarter’s traffic.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Data drift is a change in inputs, not in behavior or measurement. Nothing in your repository has to change for it to happen.
- Inputs include more than the user message: retrieved documents, tool responses, upstream generated content, and conversation shape.
- Text has no columns, so drift is measured on derived signals: embedding distributions, intent mix, language, length quantiles, retrieval scores, and unknown-class rates.
- Any drift metric needs a threshold calibrated on your own quiet periods. Numbers borrowed from someone else’s tabular pipeline will alert constantly or never.
- Watch for mix effects. When an aggregate moves while every segment holds steady, the population changed rather than the system, and the response is different.
What counts as an input
- The user message distribution. New intents, topics that spike with an external event, shorter and more elliptical phrasing as mobile share rises, machine-composed prompts arriving from a partner integration, a second language after a regional launch.
- Retrieved content. A retrieval system is partly programmed by its corpus. When a policy page is rewritten, the chunker splits it differently, or new near-duplicate pages outrank the canonical one, the context inside the prompt changed even though the template is byte-identical.
- Tool and API responses. A field renamed, an enum extended, a service returning nulls where it used to return values, a new error class from a rate limit. Each changes what the model reads mid-run.
- Upstream generated input. A summarization step, an extraction step, or another agent producing your input. When that component changes, you see drift you did not cause.
- Session shape. Turns per conversation and how much history is carried forward, which are part of the input even when individual messages look the same.
Measuring drift when the input is text
Techniques from tabular monitoring transfer only partially. Distribution distance metrics assume you can build two comparable histograms, and free text has no columns, so you build them from derived signals and accept that each is a proxy.
- Embedding distributions. Compare a recent window against a reference window using a distance measure over binned representations or cluster assignments. The most general signal and the least interpretable alone, so pair it with something readable.
- Cluster and topic mix. Cluster the embeddings and track each cluster’s share of traffic. A new cluster appearing, or an old one collapsing, is a finding you can hand to a product owner.
- Categorical proxies. Detected intent, language, client, segment, and route. Cheap frequency comparisons that usually explain what an embedding distance is reacting to.
- Numeric proxies. Input token count, turn count, and retrieval score distributions, read at several quantiles rather than as means, since the tail moves first.
- Unknown-class rate. If a router sits in front of the system, how often it falls through to a default or low-confidence branch is one of the best single indicators available, because it is the system reporting that it sees something new.
All of this comes from telemetry you already emit, which is why the span, trace, and session structure of your traces determines what you can measure. Calibrate thresholds on your own history rather than copying a generic number.
Data drift, concept drift, and why the distinction is practical
Data drift is a change in the inputs. Concept drift is a change in the relationship between input and correct output: the same question now has a different right answer because the refund window changed or the policy was rewritten. The distinction between data, concept, and model drift predates LLM applications and survives intact, because the two need different responses. Input drift is answered by extending coverage. Concept drift is answered by correcting ground truth, since your reference answers are now wrong and will penalize the system for being right.
What to do when the inputs have moved
Read the new traffic first. Filter to sessions in the shifted part of the distribution and look at them. The finding is often mundane: one large customer onboarded with a different vocabulary. Then score that slice on its own, because drift is not automatically a regression. A new intent the system handles well is a coverage gap in your test suite, not an incident.
Extend coverage deliberately. Promote examples from the new traffic into your evaluation set and re-check retrieval against the new query shapes.
Then adapt the system. Prompt, routing, and retrieval changes belong in a measured loop, as in the prompt learning playbook.
Separate mix effects from regressions. Group by segment before comparing windows.
FAQ
How is data drift different from model drift or agent drift?
Data drift is about what goes in. Model drift and agent drift are about what the system does: the model’s behavior changing, or the agent’s behavior changing across its prompts, tools, retrieval, and memory. Input drift is one common cause of behavior change, so they appear together often, but the response differs. Behavior change points you at your own dependencies. Input change points you at your traffic.
Does data drift mean quality has degraded?
Not by itself. It means the inputs are no longer the ones your system was tuned and tested against, which raises the probability of degradation without demonstrating it. Score the drifted slice on its own to find out. Plenty of drift turns out to be traffic the system handles fine and the test suite does not cover.
How do you detect drift in text inputs with no numeric features?
Derive features. Embed the inputs and compare distributions between a reference window and a recent one, cluster the embeddings and track cluster shares, and monitor readable proxies such as detected intent, language, input length quantiles, and how often a router falls through to a default branch. The embedding comparison finds that something moved; the proxies tell you what.
Can data drift be prevented?
No, and trying is the wrong goal, since users changing their behavior is usually a sign of adoption. Detect it early, keep evaluation sets fed from recent traffic so coverage follows the distribution, and treat the prompt, routing, and retrieval changes that follow as maintenance rather than emergencies.
