LLM inference is the process of running a trained large language model (LLM) to produce an output for a request. Every response an AI application or agent returns is an inference call, so inference sets the latency, cost, and reproducibility of what users see.
For example, suppose a single request takes 6.0 seconds end to end. A trace shows 1.2 seconds waiting in a queue, 0.9 seconds processing a 4,000-token prompt, 3.6 seconds generating 180 output tokens, and 0.3 seconds of delivery overhead. Each of those segments has a different fix, and switching models would not shorten the queue.
Inference is the use of a model. Training is different because it changes the model’s weights.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why LLM inference matters
Most of what you can change about an LLM application happens at inference time. Prompts, retrieved context, sampling settings, and output limits are all set per request.
Understanding inference helps with:
- Diagnosing slow responses: Find out whether the time went to queueing, prompt processing, or token generation.
- Controlling cost: See how prompt length and output length drive the cost of each request.
- Reproducing results: Record the settings that make two runs comparable.
- Debugging agents: Follow an agent task as a chain of inference requests with tool calls between them.
For a look at what inference costs at scale, see Arize’s analysis of LLM inference costs.
How LLM inference works
An inference request moves through a lifecycle, not a single step. It is queued, the prompt is processed, and tokens are generated one at a time until a stopping condition is met.
1. Arrival and queueing
The request reaches the serving system and may wait for capacity. Under load, this wait can be longer than generation itself, so it belongs in any honest latency account.
2. Tokenization and prompt assembly
The prompt is assembled from system instructions, conversation history, retrieved evidence, and the user input. It is then converted to tokens. What counts as a token depends on the model. For more, see tokenization.
3. Prefill
The model processes the whole prompt in parallel. This builds the state it needs for generation, including the key-value (KV) cache for the prompt’s tokens.
Long prompts make prefill expensive. The first output token appears when prefill finishes, so this phase drives time to first token.
Prompt caching can reuse prefill work when many requests share the same prefix. You track how often that happens with cache hit rate.
4. Decoding
The model generates one token, appends it to the sequence, and repeats. This is the next-token loop of an autoregressive model.
The spacing between generated tokens is inter-token latency. Decoding also accounts for most of the cost of long outputs.
5. Stopping and delivery
Generation ends at a stop sequence, a length limit, or a tool call. The tokens are then streamed to the client or returned all at once.
For an agent, a tool call starts the lifecycle again. The tool result comes back in, and a new inference request goes out.
NVIDIA’s NIM benchmarking guide defines serving metrics against these phases. Hugging Face’s LLM tutorial walks through the generation side.
How to break down inference latency
The following script splits the 6-second request from the opening example into phases. It also derives two rates from the token counts. It is illustrative and uses hypothetical timings.
phases = {
"queue wait": 1.2,
"prefill": 0.9,
"decoding": 3.6,
"delivery overhead": 0.3,
}
prompt_tokens = 4_000
output_tokens = 180
total = sum(phases.values())
for name, seconds in phases.items():
print(f"{name:<18} {seconds:.1f}s {seconds / total:>4.0%}")
print(f"{'end-to-end':<18} {total:.1f}s")
ttft = phases["queue wait"] + phases["prefill"]
print(f"time to first token: {ttft:.1f}s")
print(f"prefill rate: {prompt_tokens / phases['prefill']:,.0f} tokens/s")
print(f"mean inter-token latency: {phases['decoding'] / output_tokens * 1000:.0f} ms")
Output:
queue wait 1.2s 20%
prefill 0.9s 15%
decoding 3.6s 60%
delivery overhead 0.3s 5%
end-to-end 6.0s
time to first token: 2.1s
prefill rate: 4,444 tokens/s
mean inter-token latency: 20 ms
From the client’s side, the first token arrives after both the queue wait and prefill. That is 2.1 seconds here. Decoding takes 60% of the total, at an average of about 20 milliseconds per token.
The 6.0-second total is this request’s end-to-end latency. An agent task chains several such requests, with tool calls between them.
What each segment points to
Each phase suggests a different intervention:
- Queue wait: Look at capacity or routing.
- Prefill: Look at prompt length or caching.
- Decoding: Look at output length or model speed.
Without this breakdown, a team might switch models to fix what is really a queueing problem. The Phoenix documentation describes how to capture these phases as spans and attributes on traces.
What to record for each inference request
Two runs of the same model are comparable only if their inference settings match. Record these with every request, at minimum:
- The exact model version
- Temperature and related sampling settings
- Maximum output tokens
- Reasoning or effort settings, where the provider exposes them
- The tool definitions offered to the model
- The prompt or system prompt version
Also record input and output token counts, cached-token counts where reported, and timing for each phase.
Attach this metadata to the trace. Months later, “the model got worse” usually turns out to be one of these settings having changed. The same metadata is what makes evaluation results reproducible.
Limitations of measuring LLM inference
- Providers hide phases: With a hosted API, you may see only total latency and token counts. Assigning time to each phase then means inferring it from timing patterns.
- Generation is probabilistic: Because of sampling, the same prompt can produce different outputs and different lengths. Latency and cost vary across runs as a result.
- Longer context costs more: Every added token is processed in prefill and attended to during decoding. Keeping context lean is an inference-cost decision.
- Averages mislead: Queueing and decoding both have long tails. Report distributions for each phase, under a stated load.
LLM inference vs. training
Training adjusts a model’s weights on a dataset. It happens in discrete runs, such as pretraining and later fine-tuning, and each run has a large upfront cost.
Inference applies the resulting fixed weights to each new request. Its cost is paid per request, for as long as the model serves traffic. This means inference cost grows with request volume, not with dataset size.
Three practical consequences follow:
- Settings change without retraining: Model choice, output length limits, reasoning settings, and temperature are set per request.
- Bad outputs are often not model problems: Wrong retrieval, a weak prompt, or a missing tool can produce bad answers from a model that works correctly.
- Optimization targets the request: You improve inference by working on prompts, caching, batching, and model selection, not on gradients.
Frequently asked questions
What is the difference between prefill and decoding?
Prefill processes the entire prompt at once to prepare for generation, and it drives time to first token. Decoding then generates tokens one at a time. It determines inter-token latency and most of the cost of long outputs.
Is inference the same as training, just faster?
No. Training changes the model’s weights using a dataset. Inference runs the fixed model on a request and changes nothing about the model.
Why does the same prompt sometimes take much longer?
Several causes can each add seconds: queueing under load, a cache miss on the prompt prefix, or sampling that produces a longer response. A trace broken down by phase shows which one happened.
What should I record for each inference request?
Record the model version, sampling settings, output token limit, reasoning settings if exposed, and prompt version. Add input, output, and cached token counts, plus timing for each phase. That is enough to reproduce comparisons and explain changes in cost or latency.
Why does a longer prompt increase latency even with a short answer?
Every prompt token must be processed during prefill before the first output token appears. A longer prompt therefore raises time to first token, even if the answer itself is short.
Putting LLM inference to work
Trace every request with timing for each phase, not only the total. When a response is slow, check queue wait, prefill, and decoding before you change models.
Record the model version, sampling settings, output limits, and prompt version alongside token counts. These details let you compare runs and explain cost changes later.
Tools such as Arize Phoenix can capture each inference call as a span, so you can see where time and tokens went in a single request or across an agent task.
