LLM throughput is how much work a large language model (LLM) serving system completes per second. It tells you how many users or pipeline jobs your system can handle at once, which drives capacity planning, hardware choices, and serving cost.
For example, suppose you load-test the same model for 60 seconds with 32 concurrent requests. With outputs of about 100 tokens, it completes 30 requests per second and 3,000 output tokens per second. With outputs of about 800 tokens, it completes only 5 requests per second but 4,000 output tokens per second.
The server didn’t change between those two tests. Only the workload did. That is why a throughput number means little unless you also state its unit, concurrency, prompt and output lengths, and measurement window.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why LLM throughput matters
Throughput describes system capacity under load. It does not describe how fast any one user sees a response. That is latency, and the two often move in opposite directions.
Throughput is useful for:
- Capacity planning: Estimate how many servers or GPUs you need for expected traffic.
- Comparing serving setups: Test a new model, batch size, or inference server on the same workload.
- Running batch jobs: Estimate whether an offline pipeline will finish before its deadline.
- Catching regressions: See whether a model or prompt change reduced how much work the system completes.
Interactive products are usually limited by latency. Offline pipelines are usually limited by throughput, subject to a completion deadline. Decide which case you are in before you tune anything.
For more on how models generate tokens at serving time, see LLM inference.
How LLM throughput is defined
NVIDIA’s NIM benchmarking guide, LLM benchmarking metrics, defines throughput measures alongside latency measures. The two belong together in any report.
There are two common forms.
Formula:
Token throughput = output tokens generated / measurement window (s)
Request throughput = completed requests / measurement window (s)
Where:
- Output tokens generated is the total number of tokens the model produced during the window.
- Completed requests is the number of requests that finished during the window.
- Measurement window is the length of the test in seconds.
Choices you need to state
Token throughput can count output tokens only, or input plus output tokens. Conventions differ. Prefill, which processes the prompt, and decode, which generates new tokens, stress the system differently.
Also state whether the window starts at the first request or after a warm-up period. A cold start can drag the number down.
Goodput
Goodput counts only the work completed within your service-level objectives (SLOs). For example, you might count only requests that met both a time-to-first-token ceiling and a per-token latency ceiling.
NVIDIA Triton’s GenAI-Perf documentation on goodput describes computing throughput subject to such constraints.
A server can post high raw throughput by batching aggressively while every request misses its latency target. Its goodput is then low. That makes goodput a better basis for capacity decisions, as long as its thresholds match your real SLOs.
How to calculate LLM throughput
The following code calculates request and token throughput for the two load tests from the opening. It also calculates goodput for Test A under a hypothetical assumption that 1,500 of its 1,800 requests met both latency targets.
window_s = 60
tests = {
"Test A (~100-token outputs)": {"requests": 1_800, "output_tokens": 180_000},
"Test B (~800-token outputs)": {"requests": 300, "output_tokens": 240_000},
}
for name, t in tests.items():
req_per_s = t["requests"] / window_s
tok_per_s = t["output_tokens"] / window_s
print(f"{name}: {req_per_s:.0f} req/s, {tok_per_s:,.0f} output tok/s")
# Hypothetical: requests in Test A that met both latency targets.
met_slo = 1_500
goodput = met_slo / window_s
print(f"Test A goodput: {goodput:.0f} req/s ({met_slo / tests['Test A (~100-token outputs)']['requests']:.0%} of requests met the targets)")
Output:
Test A (~100-token outputs): 30 req/s, 3,000 output tok/s
Test B (~800-token outputs): 5 req/s, 4,000 output tok/s
Test A goodput: 25 req/s (83% of requests met the targets)
In this hypothetical, Test A completes 30 requests per second, but only 25 per second are useful capacity under the SLO.
Worked example: same server, different answer
Both load tests ran the same model for 60 seconds at a concurrency of 32. Only the prompt and output lengths differed.
| Metric | Test A | Test B |
|---|---|---|
| Prompt length | ~200 tokens | ~200 tokens |
| Output length | ~100 tokens | ~800 tokens |
| Requests completed | 1,800 | 300 |
| Output tokens | 180,000 | 240,000 |
| Request throughput | 30 req/s | 5 req/s |
| Token throughput | 3,000 tok/s | 4,000 tok/s |
By requests per second, Test A looks six times faster. By output tokens per second, Test B is higher.
Neither number is wrong. The tests used different workloads. A throughput claim that doesn’t state its request mix should not drive a purchasing or capacity decision.
Tokenization also matters when you compare across models, because a “token” is model-specific. For more, see tokenization.
How to measure and debug throughput
A useful load test states every condition that affects the result.
Record these for every test:
- Load: The concurrency or offered request rate.
- Workload: The prompt-length and output-length distributions.
- Sampling settings: The generation parameters used.
- Window: The measurement window and the warm-up you excluded.
- Server configuration: The batching and scheduling settings.
Report throughput next to latency percentiles and the error or timeout rate.
Two patterns help you read the results:
- Throughput flattens while latency climbs: The system is saturated, and requests are spending time in a queue.
- Throughput is low at low concurrency: The test client may be the bottleneck, or the server may not be batching enough.
Scheduling behavior such as continuous batching changes this picture, so record the server configuration with the results.
In production, capture the same measures per model and prompt version. That way a regression can be tied to a specific change. The Arize AX documentation and Phoenix documentation describe tracing instrumented LLM applications.
For more on joining traces with serving metrics, see AI observability.
LLM throughput vs. latency
Latency is per-request time. It measures how long one user waits for the first token and for the full response.
Throughput is the combined rate across all requests. A system can raise throughput by processing more requests at once while each request gets slower.
Optimizing one number in isolation is a common error. Report both at the same concurrency. For the per-request side, see time to first token.
Limitations
Throughput figures are easy to inflate and easy to misread.
- Easy to inflate: Tiny outputs, cached prompts, no latency ceiling, short windows that capture only a burst, and counting input tokens all raise the number.
- Hides fairness: One tenant’s long generations can use up the capacity another tenant is measured against.
- Says nothing about quality: A faster system that fails more tasks may cost more per successful outcome.
Report throughput with its workload, latency distribution, and a quality measure. For the quality side of a release decision, pair capacity numbers with evaluation metrics and track model performance over time, not just at launch.
Frequently asked questions
Should I report tokens per second or requests per second?
Report both if your audience makes capacity decisions. Requests per second fits costs that scale with requests. Tokens per second fits generation-heavy workloads. Either number alone hides the request mix.
What is goodput?
Goodput is the portion of throughput that meets your stated latency thresholds, and quality thresholds if you define them. It is the capacity you can actually use.
Why did throughput rise but users say the app got slower?
Higher concurrency or batching increased total output, while each request spent more time queued and generating. Check latency percentiles at the same concurrency before concluding that anything improved.
Can I compare throughput across two models?
Only on identical prompts, output lengths, and settings, with tokenization differences noted. A model that uses fewer tokens for the same answer may show lower token throughput and still finish requests sooner.
Does higher concurrency always increase throughput?
Usually up to a point. Throughput rises with concurrency until the system saturates. After that, throughput flattens while latency keeps climbing.
Putting LLM throughput to work
Every time you report throughput, include the unit, concurrency, prompt and output lengths, measurement window, warm-up, and server configuration. Add latency percentiles and the error rate.
For capacity decisions, define your latency SLOs first and measure goodput against them. Compare systems only on the same workload.
In production, trace requests per model and prompt version so you can tie a throughput or latency change to a release. Arize AX can collect those traces next to evaluation results, so speed and quality are judged together.
