Speculative decoding is a technique that speeds up text generation by letting a fast draft model guess upcoming tokens for a larger model to check. In AI engineering, it can reduce response latency without changing what the large language model (LLM) would have produced.
For example, suppose a draft model proposes five tokens: “the cat sat on the.” The target model checks all five in one pass. It accepts “the cat sat,” rejects “on,” and replaces it with “down.” That round produces four tokens for one target pass plus the draft cost, where ordinary decoding would need four target passes.
The model whose output you want is called the target model. The fast proposer is called the draft model.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why speculative decoding matters for LLM serving
Ordinary decoding is serial. The target model runs one forward pass for every token it generates, so long responses take a long time.
Speculative decoding attacks that per-token cost. It is useful for:
- Reducing latency: Cut the time per token for a single request, especially for long outputs.
- Keeping output quality: With proper verification, speed up generation without changing the target model’s output distribution.
- Evaluating serving changes: Know which metrics to check when someone turns speculation on.
- Questioning vendor claims: Ask whether a claimed speedup includes target verification and on what workload it was measured.
Speculative decoding reduces per-token latency for a request. It is not primarily a way to raise fleet throughput, and verification adds compute.
For more on how generation is served, see LLM inference.
How speculative decoding works
Speculative decoding breaks generation into propose-and-verify rounds. The Hugging Face Transformers documentation on assisted decoding describes this pattern. Each round has three steps.
- The draft model proposes γ (gamma) candidate tokens, one after another, cheaply.
- The target model scores all candidate positions in a single parallel forward pass.
- Tokens are accepted while they agree with what the target model would sample. At the first disagreement, a corrected token is drawn from the target distribution, and the next round starts.
Here is the example from above as a round:
Draft proposes: [the, cat, sat, on, the]
Target verifies: the ✓ cat ✓ sat ✓ on ✗ -> resample "down"
Accepted this round: "the cat sat down" (4 tokens for 1 target pass + draft cost)
The final “the” is discarded, because it came after the rejected token. The next round starts from “the cat sat down.”
Why the output stays the same
Speculative sampling uses a rejection-sampling rule to decide which draft tokens to accept. With that rule, the accepted tokens and the corrected token together follow the target model’s distribution exactly.
This means the draft affects speed, not what gets sampled. With greedy decoding, the output should match a target-only run token for token.
Speculative decoding builds on how autoregressive models generate one token at a time, each conditioned on all previous tokens. It changes how that computation is scheduled, not what each token is conditioned on.
The draft must share the target’s tokens
The draft model must share the target’s vocabulary and tokenization. Otherwise the target cannot check the draft’s guesses position by position.
That is why tokenization compatibility is the first thing to check when pairing models.
What determines the speedup
The realized speedup depends mainly on two things. The first is the acceptance rate, meaning how many draft tokens the target accepts per round. The second is the draft cost relative to the target.
The draft model’s standalone quality matters less than you might expect. A weaker but fast draft with high agreement beats a strong, slow draft.
Acceptance falls as the draft and target diverge on a domain. A draft trained on different data, or text in unusual formats, produces more rejections.
How to measure speculative decoding
Do not report a single speedup multiplier. Run the same prompts and sampling settings with speculation off and on, and measure each of these:
- Acceptance rate or accepted length: Track the mean number of accepted tokens per verification round. This is the main lever.
- Latency: Measure per-token latency and end-to-end latency at the percentiles your product cares about, since averages hide stalls.
- Draft overhead: Measure draft-model time and memory as a share of each round. A draft that is too large or too slow erases the gain.
- Output equivalence: With greedy decoding, outputs should match the target-only run. With sampling, compare distributions and task quality on an evaluation dataset rather than expecting identical text.
A useful report records the workload alongside the result:
target model: <model and version>
draft model: <model, or n-gram / retrieval / early-exit proposer>
gamma: <candidate tokens per round>
sampling: <greedy, or temperature and top-p>
prompts: <same prompt set for speculation off and on>
sequence lengths: <input and output token lengths>
acceptance: <mean accepted tokens per round>
latency: <per-token and end-to-end, at chosen percentiles>
draft overhead: <draft time and memory share>
quality check: <evaluation dataset result, off vs. on>
A benchmark speedup is workload-specific. Report acceptance, draft settings, and sequence lengths with it.
How to debug speculative decoding
Serving traces help attribute time between draft passes and target passes. The Arize AX documentation covers tracing LLM applications.
Watch acceptance over time. An acceptance-rate drop after a model or prompt change otherwise looks like unexplained latency. AI observability keeps that serving context visible.
After any serving change, confirm that output quality did not change. For more on that step, see LLM evaluation.
Limitations of speculative decoding
Speedup is not guaranteed, and the technique has real costs.
- Low acceptance: Code-switching text, rare formats, or a draft trained on different data can mean rounds accept only one token at a time. The system still pays draft and verification cost, so generation can end up slower than without speculation.
- Extra compute: Verification computes target outputs for positions that get discarded. Total compute per token can rise even as latency falls.
- Lower throughput under load: Under heavy batch load, that extra compute may reduce overall throughput.
- Extra memory: Memory must hold two models and their caches.
- Small gains on short outputs: Short responses have few rounds, so there is little time to save.
Treat published multipliers as results on a particular workload, and re-measure on yours.
Speculative decoding vs. approximate speedups
The dividing line is verification.
In speculative sampling, the target model checks every token, and the output distribution is preserved.
In approximate schemes, that guarantee is gone. Examples include accepting draft tokens by a loose similarity threshold, or letting the draft finish whole spans unchecked. The output distribution changes, so you must re-evaluate quality as if the model itself changed.
Distillation and quantization are different again. They change the model that produces the final tokens, not the decoding schedule.
When a vendor claims speculative decoding, ask whether target verification with a correction step is actually happening.
Frequently asked questions
Does speculative decoding change the model’s answers?
With proper speculative sampling, no. The output follows the target model’s distribution, and greedy outputs should match target-only decoding. Approximate accept rules can change answers and need quality evaluation.
What determines the speedup from speculative decoding?
Mostly how many draft tokens the target accepts per round, and how cheap the draft is relative to the target. A weaker but fast draft with high agreement beats a strong, slow draft.
Does the draft model have to be a smaller LLM?
Usually it is, but proposers can also be n-gram or retrieval-based, or early-exit heads. The requirement is fast proposals in the target’s token space that the target can verify.
Should I use speculative decoding for batch offline generation?
Measure first. Speculative decoding targets latency. At high batch sizes the GPU may already be saturated, and verification overhead can reduce throughput rather than help it.
Why is speculative decoding slower on my workload?
Acceptance is probably low. If the target rejects most draft tokens, each round produces about one token while still paying for the draft and verification. Check acceptance first, then try a draft that better matches your domain.
Putting speculative decoding to work
Before you turn speculation on, record a baseline with it off: latency at your chosen percentiles and quality on an evaluation dataset. Then turn it on with the same prompts and settings.
Record the draft model, gamma, sampling settings, sequence lengths, and acceptance rate with every result. Without them, a speedup cannot be reproduced or compared.
Keep tracking acceptance and latency in production. A drop in acceptance is often the first sign that a model or prompt change has hurt serving performance.
