Inter-token latency (ITL) is the time between one generated token arriving and the next one arriving during a streamed response. In AI engineering, ITL tells you how smoothly text flows once an answer starts, which shapes how fast a streaming chat or agent feels.
For example, suppose a streamed answer delivers 100 tokens. The first token arrives at 0.8 seconds and the last at 4.8 seconds, so the mean ITL is about 40 milliseconds (ms) per token.
That average can describe two very different experiences. Tokens might arrive steadily every 40 ms. Or 90 tokens might arrive quickly, followed by a 2-second stall in the middle of a sentence.
ITL starts after the first token appears. It says nothing about how long the user waited for that first token.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why inter-token latency matters
Users who read along as text streams notice gaps and stalls. A response can start fast and still feel slow if it pauses partway through.
ITL helps you with:
- Diagnosing slow streaming: Tell a slow start apart from slow or uneven generation after the start.
- Capacity planning: See how per-stream speed changes as concurrent traffic grows.
- Finding stalls: Spot pauses caused by serving, buffering, or orchestration between model calls.
- Comparing serving setups: Compare models, batch settings, or providers on the same measure.
Pair ITL with time to first token and end-to-end latency for a complete picture of response time.
How inter-token latency arises
During LLM inference, generation has two phases.
Prefill: The model processes the prompt and produces the first token. The wait for that token is time to first token.
Decoding: The model then produces tokens one step at a time. The interval between token n arriving and token n+1 arriving is one ITL observation.
What affects the gaps
In practice, clients often receive chunks rather than single tokens. When that happens, the gaps you measure are between chunks.
Several factors change the size of the gaps. These include batching, server load, model size, output length, and network delivery.
Definitions vary by tool
NVIDIA’s NIM benchmarking metrics and its AI Perf metrics reference define ITL among the standard serving metrics. Check the exact definition in whichever tool you use. The aggregation convention is part of the metric.
How to calculate inter-token latency
For one streamed response, start with the arrival time of each token.
Formula:
gaps: t2 - t1, t3 - t2, ..., tN - t(N-1)
mean ITL: average of those gaps
rough estimate: (tN - t1) / (N - 1)
Where:
- t1 is the arrival time of the first token.
- tN is the arrival time of the last token.
- N is the number of tokens, or chunks, in the response.
For the example above, the rough estimate is (4.8 – 0.8) / (100 – 1), which is about 40.4 ms per token.
Compare a steady stream with a stalled one
The following Python script builds two hypothetical 100-token streams with the same first and last arrival times. One is steady. The other has a 2-second stall after token 90. It then reports the mean, 95th percentile (p95), and maximum gap for each.
from statistics import mean, quantiles
def itl_stats(arrivals):
gaps = [b - a for a, b in zip(arrivals, arrivals[1:])]
p95 = quantiles(gaps, n=100)[94]
return mean(gaps), p95, max(gaps)
first, last, n = 0.8, 4.8, 100
# Even stream: 100 tokens spaced evenly from 0.8 s to 4.8 s.
even = [first + i * (last - first) / (n - 1) for i in range(n)]
# Stalled stream: 90 tokens arrive quickly, a 2-second stall,
# then the last 10 tokens. Same first and last arrival times.
fast_gap = (last - first - 2.0) / (n - 2)
stalled = [first]
for i in range(1, n):
stalled.append(stalled[-1] + (2.0 if i == 90 else fast_gap))
for name, arrivals in [("even", even), ("stalled", stalled)]:
m, p95, mx = itl_stats(arrivals)
print(f"{name:8} mean {m*1000:5.1f} ms p95 {p95*1000:6.1f} ms max {mx*1000:6.1f} ms")
Output:
even mean 40.4 ms p95 40.4 ms max 40.4 ms
stalled mean 40.4 ms p95 20.4 ms max 2000.0 ms
Both streams have the same mean ITL. In the stalled stream, the other 98 gaps shrink to about 20.4 ms to make room for the 2-second pause.
The p95 gap does not reveal the stall either. The stall is only 1 of 99 gaps in this response. The maximum gap is the number that shows it.
Across many responses, high percentiles of the pooled gaps can show stalls like this if they are common enough. Within a single response, record the maximum gap.
What to report with any ITL figure
State four things alongside the number:
- Unit: Whether gaps are per token or per delivered chunk.
- Statistic: Mean, a named percentile, or maximum gap.
- Output length: The range of output lengths measured.
- Load: The concurrency or traffic level during measurement.
Capture arrival timing on the trace span so the pattern is visible later. The Phoenix documentation covers recording token and timing attributes on LLM spans.
Inter-token latency vs. time per output token
Time per output token (TPOT) is commonly total generation time divided by the number of output tokens. It folds the whole decode phase into one average.
ITL is defined on the gaps between successive tokens. Depending on whether the first token and the total elapsed time are included, TPOT and mean ITL for the same response can differ.
Name which one you computed. How tokens are counted also matters, because token counts are model-specific. For more on that, see tokenization.
How ITL relates to other latency metrics
ITL is one part of a larger latency picture:
- Time to first token: Ends where ITL begins. A slow start with fast streaming and a fast start with stalling streaming are different problems with different fixes.
- End-to-end latency: Includes the ITL phase plus queueing, prefill, tool calls, and delivery of the complete result. A good ITL does not rescue a bad end-to-end time.
- Throughput: Counts tokens or requests per second across concurrent traffic. A server can raise throughput by batching more requests while each stream’s ITL gets worse.
- Cache effects: A high cache hit rate shortens prefill and the wait for the first token. It mostly does not change the gaps after it.
For more on the throughput side, see LLM throughput.
How to debug a poor inter-token latency
Work through likely causes in this order:
- Check load: ITL degrades as concurrent decoding requests compete. Compare ITL with the concurrency at the time.
- Check output length: Long generations can slow down as context grows.
- Check chunking in your streaming path: A proxy that buffers chunks creates gaps the model never caused.
- Check for pauses between segments: See whether stalls line up with tool calls or safety checks.
In a trace, a stall inside one model call points at serving. A stall between spans points at your orchestration. For more on reading traces like this, see distributed tracing.
Common mistakes with inter-token latency
- Reporting an unnamed figure: Per-token, per-chunk, mean, and percentile versions of ITL are all in use. An ITL number without its definition cannot be compared across tools.
- Trusting the mean: User experience during streaming is set by the worst gaps, not the average gap.
- Mixing measurement points: Network buffering and rendering on the client add gaps that a server-side measurement does not see. State where you measured.
- Judging from short outputs: With only a few tokens, one gap dominates the average. Aggregate over many responses.
Frequently asked questions
Is inter-token latency the same as time per output token?
Not always. ITL is defined on gaps between successive tokens or chunks. Time per output token usually divides total generation time by the number of output tokens, so the two can differ for the same response.
What is a good inter-token latency?
There is no universal threshold. For text read as it streams, gaps should stay below the pace at which a reader consumes text, and long stalls matter most. Set targets from your own user experience requirements and measure under realistic load.
Why does inter-token latency get worse under load?
Decoding requests share compute. As more concurrent requests are batched and stepped together, each stream receives tokens less often. This is the standard trade-off between latency and throughput in serving.
Does prompt caching improve inter-token latency?
Generally no. Prompt caching shortens prefill, which improves time to first token. ITL describes the decode phase after generation starts, which caching does not directly change.
Why doesn’t my p95 ITL show the stalls users complain about?
A single stall is a small share of the gaps in one response, so it can sit above the p95. Report the maximum gap per response, or pool gaps across many responses before you take percentiles.
Putting inter-token latency to work
Record token or chunk arrival times on each LLM span, along with output length and the load at the time. With those, you can compute mean ITL, percentiles, and maximum gap later.
When you report ITL, name the unit, the statistic, and where you measured. Report time to first token and end-to-end latency next to it.
When users say streaming feels slow, check the maximum gap per response before the mean. Tracing in Arize Phoenix can show whether a stall happened inside a model call or between spans.
