Decode is memory-bound, so checking several tokens in one pass is nearly free. Here is the mechanism, the arithmetic behind it, and why quality never moves.
Every token a language model emits costs one full forward pass. That is the shape of autoregressive generation: predict a token, append it, predict again. The dependency is strict — you cannot know token five until you have committed to token four — and for years that looked like a hard floor on generation speed.
It is not a hard floor. The limit comes from a specific hardware fact, and once you see it the way around is almost forced.
Decode is a memory problem, not a math problem
During decode, a forward pass streams the model's active weights out of high-bandwidth memory and multiplies them against a single token's worth of activations. The weights are enormous; the activations are one vector. The arithmetic units finish and wait. Decode is memory-bandwidth-bound, not compute-bound, and not by a small margin.
Put numbers on it. GLM-5.3-Flash is a mixture-of-experts model: 320B total parameters, 18B active per token, FP8 weights of roughly 328 GB. Only the active experts move per token, so call it roughly 18 GB read per pass. A 4x DGX Spark deployment at TP=4 gives roughly 1092 GB/s aggregate memory bandwidth. Divide: about 60 forward passes per second is the ceiling on a single stream, and extra FLOPS do not move it.
Now the consequence. The weights move once per pass regardless of how many token positions you push through it: five positions multiply the arithmetic by five and the memory traffic by almost nothing, so a pass over five costs roughly what a pass over one costs. If you already had five candidate tokens in hand, you could check all five in that one pass and four would be free. That idle compute is the entire opportunity; everything below is machinery for producing candidates cheap enough to be worth checking.
Draft and verify
The mechanism is four steps, repeated.
One. A cheap draft proposes. Some fast mechanism — a small model, an extra head, a lookup table — generates the next k tokens. Call k the speculation length, typically 3 to 8. It runs k sequential steps, each a small fraction of a target-model step.
Two. The target verifies in one pass. The full model takes the current context plus all k proposed tokens and runs a single forward pass over the whole sequence. Causal masking means position i sees everything before it and nothing after, so that one pass yields the target's true next-token distribution at all k+1 positions at once: what it would have said after zero drafted tokens, after one, after two, and so on.
Three. Compare and cut. Walk the proposals left to right, accepting each token the target agrees with. Stop at the first disagreement, discard everything after it, and emit the target's own token at that position instead.
Four. Repeat from the new, longer context.
Step two is where the speedup lives. Verifying k+1 positions costs about one sequential decode step, because cost is dominated by moving weights and the weights move once. You paid for one step and checked five answers.
Step three has a property that is easy to miss and load-bearing: even under total rejection you still advance. The first proposal is rejected, but the verification pass already computed the target's distribution at that position, so you sample the target's own token there and keep it — exactly one token, exactly what a plain decode step would have produced. The technique cannot go backwards in token count: worst case is parity in tokens per pass, parity plus wasted drafting time in wall clock.
Why the output is identical
This is the part most explanations bury, and it is why speculative decoding sits in a different category from every other inference optimization. The output is mathematically identical to what the target model would have produced on its own. Not close, not merely indistinguishable under evaluation. Identical in distribution, provably.
The acceptance test is not a similarity heuristic. It is rejection sampling. For a proposed token x the drafter assigned probability q(x) and the target assigns p(x). Accept x with probability min(1, p(x)/q(x)): unconditionally if the target likes it at least as much as the drafter did, otherwise at the ratio. On rejection, do not sample from p directly, which would over-represent tokens the drafter already had a chance to propose. Sample from the normalized positive part of p minus q, the corrected residual — exactly the probability mass the drafting-plus-acceptance stage failed to deliver. Accepted mass plus residual mass recovers p exactly.
The draft is therefore a proposal mechanism and nothing more. It influences how often you get a long accepted run, never which tokens are legitimate.
You cannot degrade quality by choosing a bad draft model. A poor drafter proposes tokens the target rejects, you accept fewer per pass, and you go slower — possibly slower than not speculating at all. You do not get worse text. There is no quality dial to turn the wrong way.
The guarantee holds at any temperature, not only in the greedy case where you accept while the drafted token equals the target's argmax, provided top-p and top-k are applied consistently to both distributions. Quantization, pruning and distillation trade quality for speed. This trades compute for bandwidth and leaves the output distribution untouched.
Acceptance rate is the whole economics
Define the acceptance rate, alpha, as the probability a single drafted token survives verification. Everything about whether speculation pays follows from alpha and k.
E = 1 + alpha + alpha^2 + ... + alpha^k
= (1 - alpha^(k+1)) / (1 - alpha)
E is the expected tokens per verification pass. The leading 1 is the guaranteed token from step three. Position i contributes alpha^i because it pays out only if every position before it was accepted too: acceptance is a conjunction, so value decays geometrically.
With k equal to 4: alpha of 0.3 gives E of about 1.43, alpha of 0.5 about 1.94, alpha of 0.7 about 2.77, alpha of 0.8 about 3.36.
Now subtract the cost. If a draft step costs fraction c of a target step, one cycle costs roughly 1 plus k times c target-steps. With c of 0.1 and k of 4 that is about 1.4, so speedup is E divided by 1.4: about 1.0x at alpha 0.3, 1.4x at 0.5, 2.0x at 0.7, 2.4x at 0.8. That arithmetic — not any benchmark — is the honest way to state the range: single-stream speedups around 1.5x to 3x for the acceptance rates real drafters achieve. A claim far outside it should send you back to your assumptions about c and alpha.
Chase alpha before anything else. It enters the numerator geometrically; nothing else you can tune has that leverage.
Do not speculate too far ahead. The marginal value of position k+1 is alpha^k tokens; its marginal cost is a full draft step, c. Useful speculation length is roughly where alpha^k falls to c — about k equal to 6 at alpha 0.7 and c 0.1. Past that, each position buys less than it costs and inflates the work discarded on rejection.
Keep the drafter much cheaper than the target. The k times c term grows linearly while the payoff saturates at 1/(1 - alpha). A drafter at 30 percent of target cost with k equal to 5 spends 1.5 target-steps of overhead to win a few tokens. Roughly 5 to 10 percent is where the arithmetic works.
Where drafts come from
The verification half is settled; the design space is all in the drafter.
A smaller model from the same family. The classic construction: a 1B drafting for a 70B, same tokenizer, same training distribution. It works because the two agree on the easy tokens, which are most of them. Cost: a second model to serve, load and version.
Self-speculation and early exit. Propose with a subset of the target's own layers, verify with the full stack. No second checkpoint, and the draft comes from the same parameter space, which helps agreement. Needs shallow exits trained to produce usable logits.
Multi-token prediction. Extra prediction heads trained into the model, proposing several future tokens directly. This is why MTP is increasingly designed into architectures rather than bolted on: the heads are trained jointly, so they are calibrated against the model's own distribution, they share the trunk computation, and there is no separate draft model to host or keep in sync. Drafting cost c gets small enough that the cost side of the arithmetic nearly vanishes.
MTP is not a research curiosity. Community testing reports describe a 2x DGX Spark deployment of Qwen3.8-Flash-Next reaching 900k context with vision under SGLang using MTP plus an SM121 kernel patch, stress tested at a 300k token prefill, measuring roughly 64 tok/s single stream and roughly 115 tok/s aggregate across 2 to 4 concurrent sessions. Community measurements on one configuration, not vendor specifications — but MTP is plainly what runs in production serving today.
N-gram and prompt lookup. No model at all: search the prompt and the text so far for the current suffix, and propose whatever followed it last time. Free to run, so c is essentially zero and even a mediocre alpha pays. It works startlingly well wherever output quotes input: summarization, document editing, code completion inside a known file, answers citing retrieved passages verbatim. Where the model writes genuinely new text it proposes nothing useful and costs nothing to have tried.
Multi-head and tree speculation. Medusa-style approaches attach several heads, each predicting a different future offset, so you get multiple candidate continuations rather than one line. Verification covers a tree: with a carefully built attention mask, one pass scores many branches and you keep the best-accepted path. That raises effective alpha, since a rejection on one branch may be an acceptance on a sibling, but it costs more compute per pass — the resource you have spare at batch size one and lack under load.
What speculates well
Acceptance rate is a property of the text, not of the system.
High alpha: boilerplate and scaffolding code, closing brackets and imports, schema-constrained output such as JSON, formulaic prose, legal and policy language, anything quoted or lightly edited from the prompt. Here a small drafter and the full model agree almost everywhere, and runs of five or more accepted tokens are common.
Low alpha: genuinely novel reasoning, creative prose, the first token after a branch point in a chain of thought, high-temperature sampling, factual recall the drafter simply lacks. Distributions diverge, rejection comes early, and you pay drafting cost for a token you would have gotten anyway.
Gains are therefore workload-dependent; measure on your own traffic. Code assistants, structured extraction and document editing are the best cases, open-ended reasoning the worst. Alpha also varies within a single response — high through the boilerplate, low through the part the user actually asked about.
The batching trap
Speculative decoding spends compute to save memory bandwidth. That sentence tells you both when it works and when it backfires.
At batch size one the GPU has compute to spare — that was the premise — so the trade is excellent: verification of k+1 positions is nearly free, and you convert idle arithmetic units into lower latency.
Under heavy batching the premise is gone. Large batches already give each weight load many rows of work, pushing decode back toward compute-bound. The extra verification positions and the work discarded on rejection now compete with real requests, so speculation can reduce aggregate throughput while still improving single-stream latency.
The rule, stated plainly: speculative decoding optimizes latency, and only sometimes throughput.
The Qwen3.8-Flash-Next figures sketch the shape — roughly 64 tok/s for one stream, roughly 115 tok/s summed over 2 to 4 concurrent sessions. Aggregate rises with concurrency while each individual stream slows, and the headroom speculation exploits keeps shrinking as you push further. The roughly 800 tok/s reported for GLM-5.3-Flash in a tuned serving setup sits far above the roughly 60 passes per second bandwidth allows one stream, so it is plainly an aggregate figure across concurrent work, not a latency number. Treating those two kinds of number as comparable is the most common way teams talk themselves into the wrong configuration.
Practical guidance
Enable it when you are latency-sensitive and lightly loaded: interactive chat, code completion, agent loops where something is blocked on the response. Also when your output quotes the input heavily, since prompt-lookup drafting is nearly free, or when your stack supports MTP heads the model already shipped with. Be skeptical when you are throughput-bound on a saturated fleet, when generation is open-ended and high-temperature, or when the only drafter available costs a large fraction of the target.
Measure acceptance rate first. Most stacks expose it directly. Below about 0.4 the drafter is wrong for your traffic and no tuning of k rescues it; above 0.7 you can speculate further. Track it per workload class, since an average across mixed traffic hides the variation you need to act on. Then watch single-stream time-to-last-token and aggregate tokens per second together, at your real concurrency. Pick k by starting at 3 or 4 and raising it while tokens per pass grows faster than cycle cost.
Verify equivalence once. Fix the seed, temperature and top-p, run the same prompts with speculation on and off, and diff the token IDs. They should match exactly. If they do not, you have a real bug: a mismatched tokenizer, sampling parameters applied inconsistently to draft and target, a KV-cache rollback that does not truncate on rejection, or a non-deterministic kernel. The theory guarantees the match, so a mismatch is always an implementation defect.
Checklist
- Confirm decode is bandwidth-bound: active weight bytes per token over memory bandwidth is your single-stream ceiling.
- Use a drafter costing roughly 5 to 10 percent of the target, or MTP heads if the model ships with them.
- Start at speculation length 3 or 4; raise k only while alpha^k stays above the draft cost fraction.
- Measure acceptance rate before anything else; below 0.4, change the drafter rather than tuning k.
- Track single-stream latency and aggregate throughput separately, at real concurrency, and shrink or disable speculation above a load threshold.
- Diff token IDs with speculation on and off at a fixed seed. Any difference is a bug in your implementation, never a property of the technique.