A quantile is a cutpoint that divides a distribution into intervals containing equal shares of the data. The median is the most familiar one: it is the 0.5 quantile, the value that half the observations fall below. Split the same distribution at 0.25 and 0.75 and you get quartiles. Split it into 100 parts and you get percentiles.
Quantiles matter in production machine learning because they describe a distribution without assuming it has a convenient shape. Means and standard deviations quietly assume something close to a normal distribution. Latency, token counts, retrieval scores, and eval scores are almost never normal. They are skewed, they have long tails, and the tail is usually the part that hurts. A quantile lets you talk about that tail directly.
Key takeaways
- A q-quantile splits a distribution into q intervals of equal probability. Quartiles, deciles, and percentiles are all special cases of the same idea.
- Quantiles describe skewed data honestly, which is why p95 and p99 latency are reported instead of average latency.
- Quantile bounds such as p1 and p99 are a standard way to define what “in range” means for a feature or an input, and to alert when new data falls outside it.
- Quantile binning is how many drift metrics divide a continuous variable into comparable buckets before measuring distance between two distributions.
- Quantiles cannot be averaged. The p99 of a combined population is not the mean of each population’s p99.
How quantiles are defined
For a value q between 0 and 1, the q-quantile is the smallest value x where at least a q fraction of the data is less than or equal to x. Setting q to 0.5 gives the median. Setting it to 0.99 gives the value that 99% of observations fall below.
The named families are just conventions for how many cuts you make:
- Quartiles cut the distribution into four parts at 0.25, 0.5, and 0.75.
- Deciles cut it into 10 parts at 0.1 through 0.9.
- Percentiles cut it into 100 parts, which is where p50, p95, and p99 come from.
One detail trips people up: with a finite sample, a quantile often falls between two observed values, and different libraries interpolate differently. NumPy, Pandas, and most database engines each expose several interpolation methods. If two systems report slightly different p99 values for the same data, mismatched interpolation is usually the reason before anything more interesting is.
Why quantiles show up everywhere in production AI
Latency and cost. Average latency hides the requests that made users leave. A model serving p50 in 200 milliseconds and p99 in 14 seconds is two different products depending on which request you get. The same logic applies to token spend per request, where a small number of runaway agent loops can dominate the bill.
Data quality bounds. Quantiles are a practical way to define the normal operating range of an input without hand-writing thresholds. Capturing p1 and p99 from a reference window gives you an out-of-range check that adapts to the actual data rather than to someone’s guess from six months ago. This is one of the standard checks in data quality monitoring, alongside missingness and type mismatch.
Drift detection. Most distribution distance metrics need two comparable histograms before they can measure anything. For a continuous variable, quantile binning is a common way to build those histograms, because equal-frequency bins keep each bucket populated even when the variable is heavily skewed. Once the bins exist, the distance calculation can run.
Eval score distributions. LLM and agent evaluations produce score distributions, not single numbers. A mean eval score of 0.82 can describe a system that is uniformly mediocre or one that is excellent most of the time and fails badly on a specific slice. Looking at the low-end quantiles of the score distribution is how you tell those apart, which is a routine step when you are tracing and evaluating an agent rather than looking at a single aggregate.
Where quantiles mislead
The most common mistake is averaging them. If service A has a p99 of 100 milliseconds and service B has a p99 of 300 milliseconds, the p99 of traffic across both is not 200 milliseconds. Percentiles have to be recomputed from the combined data, or estimated with a structure built for the job such as a t-digest or an HDR histogram.
The second is reading extreme quantiles from small samples. A p99 computed from 50 observations is being estimated from roughly one data point. It will move a lot between windows and it will generate alerts that mean nothing. If you are monitoring a low-traffic route, either widen the window or monitor a less extreme quantile.
The third is treating a stable quantile as evidence that nothing changed. Two very different distributions can share a median. Quantiles summarize, and every summary discards information, which is why distribution monitoring usually tracks several cutpoints rather than one. The same caution carries directly into agent observability where the interesting failures live in the tail of a session distribution rather than in the average.
FAQ
What is the difference between a quantile and a percentile?
A percentile is a quantile expressed on a 0 to 100 scale instead of a 0 to 1 scale. The 0.95 quantile and the 95th percentile are the same cutpoint. Percentile is the more common word in engineering contexts, quantile is the more common word in statistics.
What is the difference between a quantile and a quartile?
A quartile is one specific set of quantiles: the three cutpoints at 0.25, 0.5, and 0.75 that divide data into four equal groups. Every quartile is a quantile. Most quantiles are not quartiles.
Why do teams report p99 latency instead of average latency?
Because the average is dominated by the common case and the tail is where users experience failure. A system can hold a good average while a meaningful share of requests time out. The p99 makes that visible, and it is the number that maps to what a frustrated user actually experienced.
How many quantile bins should I use for drift detection?
There is no universal answer, but 10 to 20 equal-frequency bins is a common starting point for a continuous feature. Too few bins hides real shifts inside a wide bucket. Too many bins makes each bucket sparse, which makes the distance metric noisy and sensitive to sample size. The right number depends on how much traffic the feature sees per monitoring window.
Can I use quantiles on categorical data?
Not directly. Quantiles require an ordering, so they apply to numeric or ordinal variables. For categorical variables, distribution comparison uses category frequencies rather than cutpoints.
