Time to first token (TTFT) is the time between a request reaching a language model service and the first response token reaching the caller. In AI applications, TTFT decides how long a user waits before a streaming answer starts to appear, so it shapes whether a chat interface or agent feels responsive.
For example, suppose a support assistant sends a 6,000-token prompt and the client measures 1.8 seconds to the first token. Server timing shows 900 milliseconds in a queue, 700 milliseconds of prompt processing, and 50 milliseconds to emit the token. The remaining 150 milliseconds is network and client handling. Half of the wait happened before the model started work.
TTFT covers the wait before generation is visible. It does not include the time spent generating the rest of the response.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why time to first token matters
A slow first token makes an application feel stuck, even when the full answer finishes on time. Users often retry, abandon the request, or send a duplicate message when nothing appears.
TTFT is useful for:
- Judging responsiveness: Measure how long users wait before they see any progress in a streaming interface.
- Locating delay: Split the wait into queueing, prompt processing, and delivery so you fix the right stage.
- Catching context regressions: Spot when a larger prompt, more retrieved text, or extra tool definitions slow down the start of a response.
- Comparing releases and providers: Check whether a model, region, or serving change made the first token arrive sooner or later.
- Setting service objectives: Track tail TTFT for interactive paths, where slow outliers drive user complaints.
For the wider set of signals around a request, see AI observability.
How time to first token works
A request passes through several stages before any text becomes visible. Each stage can dominate TTFT, so a single number does not tell you the cause.
Stage 1: Queueing
The request may wait in a queue while the serving system assigns capacity. That wait depends on traffic, priority, batching choices, and available accelerators.
A system can run a fast model and still show poor TTFT when requests wait for admission. Scheduling techniques such as continuous batching change how long that wait is under load.
Stage 2: Prefill
Next, the model processes the prompt in a phase commonly called prefill. It reads the system instructions, conversation history, retrieved context, tool definitions, and user input.
During prefill, the model builds the internal state it needs to predict the first output token. Prefill work grows with input length. This is one reason two requests to the same model can have very different TTFT values.
Caching and serving design can change how much prefill work is repeated.
Stage 3: First token and delivery
Only after prefill does the model produce and return the first token. Network transfer, proxy buffering, and client rendering can add further delay before the user sees it.
Where timing starts and stops
Some dashboards define TTFT at the server boundary. Others define it as the moment the client receives the token. Those definitions are not interchangeable.
NVIDIA’s LLM metrics documentation and its AI performance metrics reference describe TTFT alongside other serving measures. Treat them as the primary source for the serving-side definition. Then state any local variation in your own dashboard.
How to measure time to first token
Record a timestamp at each stage boundary. If your platform exposes only one TTFT field, document where its timing starts and stops.
A useful trace record keeps the components separate:
request_received -> queue_start -> prefill_start -> first_token_emitted -> first_token_received
Formula:
ttft_server = first_token_emitted - request_received
ttft_client = first_token_received - request_received
Where:
- request_received is when the service accepts the request.
- first_token_emitted is when the server sends the first output token.
- first_token_received is when the client receives that token.
- ttft_server is TTFT measured at the server boundary.
- ttft_client is TTFT as the caller experiences it.
Calculate a TTFT breakdown in Python
The following script uses the support assistant timings from the opening example. It computes both TTFT definitions and shows how much of the client wait each stage accounts for.
events = [
("request_received", 0),
("prefill_start", 900),
("prefill_end", 1600),
("first_token_emitted", 1650),
("first_token_received", 1800),
]
labels = ["queue", "prefill", "emit first token", "network and client"]
t = dict(events)
ttft_server = t["first_token_emitted"] - t["request_received"]
ttft_client = t["first_token_received"] - t["request_received"]
print(f"server TTFT: {ttft_server} ms")
print(f"client TTFT: {ttft_client} ms")
for label, (start, end) in zip(labels, zip(events, events[1:])):
ms = end[1] - start[1]
print(f"{label:<20}{ms:>5} ms {ms / ttft_client:>6.1%}")
Output:
server TTFT: 1650 ms
client TTFT: 1800 ms
queue 900 ms 50.0%
prefill 700 ms 38.9%
emit first token 50 ms 2.8%
network and client 150 ms 8.3%
The server and client values differ by 150 milliseconds. A team comparing one system’s server TTFT with another system’s client TTFT would be comparing different things.
Keep the prompt-token count and finish reason beside those timestamps. Without them, a TTFT regression after a context change looks like a model regression.
Worked example: what to fix first
Here is the same 1.8-second wait, laid out by stage.
| Stage | Time | Share of client TTFT | What changes it |
|---|---|---|---|
| Queue | 900 ms | 50.0% | Capacity, scheduling, priority |
| Prefill | 700 ms | 38.9% | Prompt length, caching |
| Emit first token | 50 ms | 2.8% | Model and serving stack |
| Network and client | 150 ms | 8.3% | Proxies, buffering, rendering |
Shortening the prompt might save part of the 700 milliseconds of prefill. It will not fix the queue.
Adding capacity or changing scheduling addresses the larger share. This breakdown prevents a common mistake: blaming the model for delay that occurred before the model started work.
How to debug a slow time to first token
Start with percentiles rather than the mean. A p50 (median) TTFT of 400 milliseconds can coexist with a p95 of several seconds during traffic spikes.
For a chat interface, the tail often determines whether users retry or give up. For an agent, TTFT matters most at interactive checkpoints. A background batch job may tolerate a longer wait if total throughput and cost are acceptable.
Split traces by model, prompt-token band, region, cache state, and concurrency. This keeps a healthy median from hiding a slow class of requests.
Then use tracing to place the delay:
- Queue time rises with concurrency: The problem is capacity or scheduling.
- Prefill rises with input tokens: Inspect context size, repeated instructions, and tool schemas.
- Server TTFT is stable but client TTFT is not: Inspect streaming proxies, buffering, and rendering.
A trace connects each model call’s timing to the retrieval, tool, and application steps around it. The Arize Phoenix documentation and Arize AX documentation cover tracing workflows for this kind of breakdown.
What time to first token cannot tell you
TTFT is one responsiveness measure. Several things that matter to users sit outside it.
- Answer quality: A fast first token can be followed by a wrong answer, a stalled stream, or a failed tool call.
- Total wait in multi-step agents: Several model calls, retrievals, and tools may run before a visible result. Measure time to first visible progress and total task time as well as each call’s TTFT.
- Comparable numbers across systems: Some providers include queueing and some start timing after admission. Some report the first non-empty chunk rather than the first token.
- Exact token boundaries: Streaming chunks can contain multiple tokens, and a chunk boundary is not always a token boundary.
Reducing TTFT can also create tradeoffs. Smaller context may omit evidence the model needs. Batching changes can lower throughput. Speculative or smaller-model shortcuts need quality checks.
Treat TTFT as one service objective among correctness, cost, and completion time. For where latency sits among task-level measures, see agent evaluation metrics.
Time to first token vs. end-to-end latency and throughput
TTFT ends at the first token. End-to-end latency ends when the response or task is complete.
A model can start quickly and finish slowly, or start slowly and then stream quickly. Inter-token latency describes the generation phase after the first token. Do not fold it into TTFT without saying so.
Throughput describes how much work a system completes per unit of time across many requests. Optimizing for large batches can raise throughput while making an individual request wait longer.
Interactive products usually watch TTFT and tail latency. Offline evaluation or document processing may accept a higher TTFT in exchange for throughput and lower cost.
Frequently asked questions
Is a lower time to first token always better?
Lower is usually better for interactive use because the user sees progress sooner. It is not enough on its own. Check completion time, later-token speed, cost, and answer quality. A change that starts faster but causes more retries can make the task slower overall.
Does prompt length affect time to first token?
Yes. More input tokens generally mean more prefill work before the first output token. The size of the effect depends on serving design and whether repeated prompt content can be reused. Measure TTFT by prompt-token band instead of assuming one effect for every request.
Why is my p95 time to first token much worse than the median?
The tail usually reflects queueing under load, unusually long prompts, cold capacity, or network and proxy delays. Break the p95 down by those dimensions. Averages and medians alone hide the requests users actually complain about.
Should agents report time to first token for every model call?
Report it for calls on the user-visible path. Keep total task latency as the primary measure for multi-step work. Per-call TTFT helps you find a slow step, and task latency tells you what the user experienced.
Is server TTFT or client TTFT the right number?
Use both if you can, and label each one. Server TTFT isolates the serving system. Client TTFT includes network and rendering delay, which is what the user actually feels.
Putting time to first token to work
Record the timestamps for each stage, not only a single TTFT value. Store the prompt-token count, cache state, region, and concurrency with each request.
Report TTFT at p50 and p95, and say whether it is measured at the server or the client. Track later-token speed and total task time separately.
When TTFT rises, find the stage that grew before changing the prompt or model. Tools such as Arize Phoenix can capture model-call spans so you can see whether the time went to queueing, prefill, or delivery.
