Continuous batching is a way of serving a large language model (LLM) where requests join and leave the running batch at every generation step. It matters because it keeps GPUs busy under mixed request lengths, which raises throughput and cuts the time short requests spend waiting behind long ones.
For example, suppose a server runs eight requests together. Seven need 20 output tokens and one needs 400. With static batching, all eight slots stay occupied for 400 steps, and only 540 of the 3,200 slot-steps do useful work. With continuous batching, the seven short requests leave after step 20 and queued requests take their slots right away.
Continuous batching is also called iteration-level batching. The model does not change. Only the scheduling does.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why continuous batching matters
Interactive LLM traffic rarely arrives in neat groups of equal length. Some users want a one-line answer and others want a long report.
When a server waits for a whole batch to finish, short requests pay for the long ones. Continuous batching removes that wait.
Continuous batching is useful for:
- Raising throughput: Keep batch slots filled with active requests instead of finished ones.
- Reducing wait for short requests: Return each request as soon as it finishes and admit the next one.
- Serving live traffic: Handle requests that arrive at different times with different output lengths.
- Comparing serving setups: Understand why two servers running the same model report very different throughput and latency.
How continuous batching works
Autoregressive generation runs in steps. First, a prefill phase processes the prompt. Then, during decoding, the model produces roughly one token per request per step.
Because autoregressive models generate token by token, a server can change which requests are in the batch between steps. The Hugging Face Transformers documentation on continuous batching describes scheduling at this step-level granularity.
Static batching
With static batching, the server assembles a group of requests and runs them together. It returns results only when the slowest request finishes its last token.
Consider the opening example. If seven requests need 20 tokens and one needs 400, seven slots sit occupied doing padded or idle work for most of the run.
Continuous batching
With continuous batching, the scheduler re-forms the active set at every step. Finished sequences are returned as soon as they end. Waiting requests are admitted as capacity allows.
Step t: [A, B, C, D] B finishes -> return B immediately
Step t+1: [A, C, D, E] E (queued) joins after its prefill
Step t+2: [A, C, D, E] ... until each sequence completes
Here, request B leaves the moment it finishes. Request E, which was waiting in the queue, takes the free slot after its prefill runs.
What still limits admission
A free slot does not always mean a new request can start. The server also needs memory in its key-value (KV) cache to hold the new request’s state.
When the cache is full, requests may be delayed, preempted, or recomputed. Long prompts can also stall other requests during prefill unless the server splits prefill work into chunks. For more on that memory, see KV cache.
Simulate static and continuous batching in Python
The following script compares the two schedules in a simplified model. It uses the opening example plus eight more 20-token requests waiting in the queue, all hypothetical.
The model ignores prefill and treats every decode step as equal cost. Real servers do not behave this simply, so read the output as an illustration of scheduling, not a benchmark.
from collections import deque
BATCH_SIZE = 8
first_wave = [400] + [20] * 7 # one long request, seven short ones
waiting = [20] * 8 # eight short requests already queued
def static_batching(requests):
finish, step = [], 0
for i in range(0, len(requests), BATCH_SIZE):
batch = requests[i:i + BATCH_SIZE]
step += max(batch) # the batch waits for its longest request
finish += [step] * len(batch)
return finish
def continuous_batching(requests):
queue = deque(enumerate(requests))
active, finish, step = {}, [None] * len(requests), 0
while queue or active:
while queue and len(active) < BATCH_SIZE:
idx, length = queue.popleft()
active[idx] = length
step += 1 # one decode step for every active request
for idx in list(active):
active[idx] -= 1
if active[idx] == 0:
finish[idx] = step
del active[idx]
return finish
requests = first_wave + waiting
for name, fn in [("static", static_batching), ("continuous", continuous_batching)]:
done = fn(requests)
queued = done[len(first_wave):]
print(f"{name:<11} all done at step {max(done)}, "
f"queued requests finish on average at step {sum(queued) / len(queued):.1f}")
useful = sum(first_wave)
held = BATCH_SIZE * max(first_wave)
print(f"static first batch: {useful} useful slot-steps of {held} ({useful / held:.1%})")
Output:
static all done at step 420, queued requests finish on average at step 420.0
continuous all done at step 400, queued requests finish on average at step 42.5
static first batch: 540 useful slot-steps of 3200 (16.9%)
Worked example: where the queued requests finish
Here is what happens to the eight queued requests in the simulation above.
| Schedule | When queued requests start | Average finish step for queued requests | All 16 requests done |
|---|---|---|---|
| Static batching | After the 400-token request ends | 420.0 | Step 420 |
| Continuous batching | As soon as a slot frees up | 42.5 | Step 400 |
Under static batching, the queued requests wait 400 steps for the long request, then take 20 more steps each. They all finish at step 420.
Under continuous batching, seven queued requests take the free slots after step 20 and finish at step 40. The eighth starts once a slot frees up again and finishes at step 60. Their average is 42.5.
The 400-token request finishes at step 400 in both cases. Continuous batching does not make a long generation faster. It reduces wasted capacity around it.
How to measure continuous batching
Compare serving configurations at the same concurrency, prompt-length mix, and output-length mix. Report more than one number.
- Throughput: Completed tokens or requests per second over the measurement window. For more, see LLM throughput.
- Latency distribution: Time to first token and per-token latency at the median and at high percentiles, not only the mean. Define where timing starts and stops before measuring.
- Queue time: How long requests wait before prefill. Rising queue time with flat throughput usually means the server is saturated, not inefficient.
- Batch occupancy and KV cache pressure: How full the active batch is, and whether requests are delayed, preempted, or recomputed because cache space ran out.
How to debug latency with continuous batching
Instrument the client and, where the server exposes it, the scheduler. Each view answers a different question.
Application traces show timing from the caller’s side. The Arize AX documentation and Arize Phoenix documentation cover tracing the served application.
Server metrics explain whether time went to queueing, prefill, or decode. Relate the two views using AI observability practices. Do not treat one request’s latency as a server benchmark.
When a request passes through several services before reaching the model, keep the full path in view with distributed tracing.
Limitations of continuous batching
Continuous batching removes one source of waiting. Several others remain.
- Long generations: An individual long request takes as many steps as before.
- Prefill stalls: Very long prompts can delay other requests during prefill unless the server chunks prefill work.
- Admission fairness: Admission policies decide which tenant or request class gets slots first, and that is not always fair.
- Overload: Throughput may hold while high-percentile latency climbs. A fuller batch is not automatically a healthier service.
- Portability: Framework, scheduling policy, maximum batch size, and cache limits all change results. Pin them when you report a comparison.
Continuous batching vs. static batching and Batch APIs
Static batching is simpler. It can be entirely adequate offline, where requests arrive together and nobody waits interactively. Continuous batching suits live traffic with varied generation lengths.
A provider Batch API is a different product. You submit a bulk job, accept a long completion window, and receive results later. It optimizes cost for asynchronous work and involves no step-level scheduling of your live requests.
Confusing these leads to bad expectations. Enabling a Batch API will not improve interactive latency. Continuous batching will not give you bulk-job pricing.
Frequently asked questions
Does continuous batching change model output?
No. It changes when requests run together, not the computation for a given request. Outputs should match for the same model and decoding settings, apart from ordinary numerical variation in batched execution.
Why is tail latency still poor with continuous batching?
A request can still wait in the queue, wait behind long prefills, or compete with many concurrent decodes. Continuous batching removes the static-batch wait, which is only one cause.
Is continuous batching the same as a Batch API?
No. Continuous batching schedules live requests inside a server at each generation step. A Batch API accepts bulk jobs for later, asynchronous completion.
Does continuous batching make a single request faster?
Not on its own. A long generation needs the same number of decode steps. The gain comes from not making short requests wait for it.
What should I report when comparing LLM servers?
Report concurrency, prompt and output length distributions, throughput, time to first token, per-token latency percentiles, and the scheduler and cache limits you used. A single tokens-per-second figure hides the tradeoffs.
Putting continuous batching to work
Before you compare servers, fix the traffic mix you test with. Use the same concurrency, prompt lengths, and output lengths for every configuration.
Record throughput, queue time, time to first token, and per-token latency at p50 and p95. Note the framework, scheduling policy, maximum batch size, and cache limits next to the results.
When latency rises under load, check queue time and KV cache pressure before blaming the model. Tracing your application with Arize Phoenix shows the caller’s view, which you can line up against the server’s own scheduler metrics.
