Speculative Decoding: Faster Inference Without Changing the Output

Generating a token requires reading every weight in the model. That is the whole problem: a single forward pass moves gigabytes of parameters through memory to produce roughly 2 bytes of output, so decoding is bound by memory bandwidth rather than arithmetic. Speculative decoding exploits that imbalance. A small draft model guesses several tokens ahead, the big model checks all of them in 1 pass, and every guess that survives was effectively free.

What is speculative decoding actually doing?#

It converts sequential work into parallel verification. A cheap drafter proposes the next few tokens one at a time, then the expensive model evaluates that whole proposed sequence in a single forward pass. Accepted tokens cost almost nothing extra, because verifying 4 tokens costs about what generating 1 does.

The target model's single forward pass is the expensive part, and it costs the same whether it checks 1 token or 5.

The economics come entirely from the memory-bandwidth bottleneck. Loading the target model's weights dominates the cost of a forward pass, and that load happens once whether you are scoring 1 token or 5. So the marginal cost of checking extra candidate tokens is close to zero, and every candidate that matches what the model would have produced anyway is a token you did not have to run a separate pass for.

When a guess is wrong, you discard it and everything after it, then continue from the last accepted position. That is why the drafter's accuracy, not its speed, is what mostly determines the payoff. A fast drafter that is usually wrong throws away most of its work and adds its own runtime on top.

It helps to see why the rejection rule has to be strict. The naive version of this idea would be to accept a draft token whenever the target model also considers it reasonably likely, which sounds harmless and quietly changes what the model produces: over thousands of tokens you would drift toward whatever the drafter prefers. The actual acceptance test compares the 2 models' probabilities for the proposed token and, when it rejects, resamples from a corrected distribution built from the difference between them. That correction is the part that makes the output statistically indistinguishable from having run the big model alone, and it is why speculative decoding is a genuine free lunch rather than an approximation people have agreed not to look at closely.

Why is this not a quality tradeoff?#

Because the verification step decides the output, not the draft. The target model checks each proposed token against what it would have sampled itself, and the acceptance rule is constructed so the surviving sequence follows the target model's distribution exactly. The drafter influences speed, never content.

That guarantee is what separates speculative decoding from every other inference optimization on this list. Quantization changes the weights and accepts some quality loss for memory. Distillation trains a different, smaller model. Speculative decoding leaves the model untouched and only changes the order in which work happens, which is why it composes cleanly with quantization rather than competing with it.

One honest caveat belongs next to the theory. vLLM's documentation notes that while the approach aims for theoretical and algorithmic losslessness, outputs can still vary because of floating-point precision differences and batch-size-dependent numerical stability. Source So token-for-token reproducibility against a non-speculative run is not something to promise, even though there is no systematic quality penalty.

What decides whether it actually pays off?#

Acceptance rate, and it falls fast as you speculate further ahead. Each additional draft token is a compounding bet: token 4 only helps if tokens 1, 2, and 3 were all accepted first. Past a few tokens the odds collapse and the extra drafting is wasted work.

Measured numbers make the shape concrete. Testing gpt-oss-120B with an EAGLE3 draft head on an H200, acceptance ran at 45.4% with 2 draft tokens, 35.6% with 3, and 28.3% with 4. Source

Measured on gpt-oss-120B with an EAGLE3 draft head on an H200. Each extra token compounds the odds downward.

Acceptance also depends on the text itself, which is why the same configuration behaves differently across workloads. Predictable, templated output such as code with heavy boilerplate or structured formats drafts well, since the next token is often nearly determined. Creative or high-entropy generation drafts badly. Raising sampling temperature widens the target model's distribution and makes agreement less likely, so a setting you changed for output variety quietly reduces your speedup too.

Does it still help under heavy load?#

The received wisdom says no, and recent measurement disagrees. The theory is that at high concurrency the GPU is already compute-saturated, so spare capacity for speculation disappears and the drafter becomes pure overhead. That reasoning is sound, and it is not the whole story on current hardware.

From 2026 vLLM testing on gpt-oss-120B. Vendor-published, on a strong draft head and high-bandwidth hardware.

The 2026 vLLM testing on gpt-oss-120B explicitly challenges the conventional view, reporting that gains persisted across every level tested from 1 up to 200 concurrent requests. On a conversational ShareGPT workload, peak output throughput reached 2,574 tokens per second against a 2,024 baseline, a 27.2% peak improvement and a 20.7% geometric mean. On SWE-bench code traffic, the peak gain was 24% at concurrency 100, with a 20.5% geometric mean and cost per 1 million output tokens falling 19.4%. Latency moved too: request latency down 15.9 to 20.3% across workloads, and P95 inter-token latency down 12.4% on ShareGPT.

Read that carefully rather than as a blanket reversal. Those results come from a strong modern draft head on a high-bandwidth accelerator, and vendors publish the configurations that work. vLLM's own guidance is that real gains depend on your model family, traffic pattern, hardware, and sampling settings, and it recommends reproducible benchmarking in your environment rather than trusting a table. The safe reading is that "turn it off above batch size 32" is outdated as a universal rule and still correct for some setups, so the batch-size threshold is something to measure rather than inherit.

Which drafting method should you pick?#

vLLM ships 8, and they trade setup effort against gain. N-gram and suffix decoding need no extra model and are trivial to switch on. EAGLE and multi-token prediction need a trained draft head or native model support and deliver more. A separate draft model sits in between.

Adoption cost is the real axis. Start where the setup is a flag, escalate only if acceptance looks healthy.

vLLM's own comparison splits the recommendation by traffic shape. Under low QPS, where latency is what you care about, the model-based methods such as EAGLE, MTP, and draft models deliver high gains while the lightweight n-gram and suffix approaches give low to medium. Under high QPS, where throughput matters, EAGLE and PARD land in the medium-to-high band and most other methods sit at medium.

The practical ordering follows from cost of adoption. N-gram speculation is a configuration flag, costs nothing, and is worth trying first purely to establish whether your workload drafts well at all. If acceptance looks promising, a trained draft head is where the real gains are. If your target model has native multi-token prediction support, use it, since it avoids maintaining a second model entirely. Keeping a separate draft model in sync with the target is real operational weight, which is the same reason running a smaller model outright sometimes beats accelerating a big one.

How should you decide?#

Treat it as a benchmark question, not a design question. Speculative decoding does not change your model, your prompts, or your outputs, so the only thing to establish is whether it is faster on your traffic at your concurrency. That makes it unusually cheap to evaluate and unusually hard to reason about from first principles.

Acceptance rate is the gate. Everything downstream is tuning, and everything should be measured at real concurrency.

Start with the cheapest drafter your stack supports and measure acceptance rate first, because that single number predicts everything else. If acceptance is healthy, tune draft depth and re-measure at production concurrency rather than at batch size 1, which is where every vendor benchmark looks best. If acceptance is poor, the honest conclusion is that your workload is too high-entropy to draft well, and no amount of tuning fixes that.

It is also worth being clear about what problem this solves. Speculative decoding attacks decoding speed for a single stream, so it shows up as better inter-token latency and a more responsive feel, which matters most for streaming interfaces. It does nothing for time to first token beyond second-order effects, does not reduce memory footprint the way quantization does, and does not make a self-hosted deployment cheaper than an API if the underlying breakeven math never worked. It is a genuine speedup with an unusually clean guarantee attached, applied to 1 specific bottleneck.