Local LLM throughput on a single user is dominated by memory bandwidth, not by raw compute (local AI inference speed). Speculative decoding is a technique that can break that ceiling without new silicon: it makes a small model propose a few tokens, the big model verifies them all in one parallel pass, and the wall-clock per token drops.
It is also the technique with the most options, the most caveats, and the most partial ship status across the runners people run at home.
How speculative decoding works
Generation is serial because every new token depends on the one before it. Speculative decoding breaks that by letting a cheap proposer generate a short block of candidate tokens, then having the target model score them all in a single batched forward pass. The target’s logits accept the longest prefix where proposer and target agree, and the rest is discarded.
The win is the asymmetry between the two costs. llama.cpp’s docs describe the underlying observation: “computing n tokens in a batch (as in prompt processing) is more efficient than computing n sequentially (as in response generation)” (llama.cpp speculative decoding documentation). Generation throughput is memory-bandwidth-bound; verification is compute-bound. If the proposer is right often enough, you spend the cheap wall time to save the expensive wall time.
How often the proposer is right is the acceptance rate, and it is what sets the speedup. All three published figures below are vendor- or author-reported, not independent measurements:
| Source | Setup | Vanilla | With speculation | Speedup |
|---|---|---|---|---|
| AngelSlim/Qwen3-4B_eagle3 (HF card) | Qwen3-4B, vLLM 0.11.2, H20, GSM8K / Alpaca / HumanEval / MT-bench mean | 233.26 t/s | 386.94 t/s (EAGLE-3, 2.08 accept length) | 1.66x |
| yuhuili/EAGLE3-LLaMA3.1-Instruct-8B (HF card) | 13B target, original EAGLE evaluation | (vanilla baseline) | 5.6x (EAGLE-3) / 4x (EAGLE-2) / 3x (EAGLE-1) | up to 5.6x |
| EAGLE-3 paper (arXiv:2503.01840) | SGLang, batch size 64 | (baseline) | (EAGLE-3) | 1.38x throughput |
The headline figures measure different things. The 5.6x on the yuhuili card is the EAGLE authors’ own figure for a 13B target, and the 6.5x in the EAGLE-3 abstract is a best case across its five tasks. AngelSlim summarises its own Qwen3 EAGLE-3 heads on vLLM as reaching accept lengths of 1.8 to 3.5 and maximum speedups of 1.4x to 1.9x. The 1.66x (computed from the card’s mean throughput figures) on a 4B Qwen at batch size 1 is a more conservative planning figure.
What runners actually ship
llama.cpp exposes ten speculative strategies through --spec-type. vLLM exposes a parallel set through speculative_config. They are not the same list, and the same name can mean different code in each runner.
--spec-type (llama.cpp) | vLLM method | Draft model needed? | Where it shines |
|---|---|---|---|
draft-simple | draft_model | yes | long, repetitive generation with a same-family smaller model |
draft-eagle3 | eagle3 | yes (EAGLE-3 trained) | high acceptance across diverse prompts |
draft-dflash | dflash | yes (DFlash trained) | single block per forward pass, GPU-friendly drafting |
draft-dspark | not in the docs’ common method values | yes (DSpark trained) | Qwen3 backbones only at last check |
draft-mtp | mtp | uses target’s MTP heads | Gemma 4 and models that ship native MTP |
ngram-cache, ngram-simple, ngram-map-k, ngram-map-k4v | ngram | no | code refactoring, repeated boilerplate, summarisation |
ngram-mod | no direct equivalent listed | no | reasoning models that loop back into prior text |
The five non-trivial strategies each have a real tradeoff and a published catch.
draft-simple (a.k.a. plain draft model). A smaller same-family model generates drafts, and the target verifies them. The acceptance is highest when the draft and target share a tokenizer and have similar training. llama.cpp’s docs call this “the most used approach in speculative decoding” (llama.cpp speculative docs). The catch is the extra VRAM: the draft has to fit alongside the target, and it has to come from the target’s family. A small Qwen drafting for a large Qwen is the shape to aim for; pairing a 1B Llama with a 70B Qwen is not.
draft-eagle3. EAGLE-3 reads the target’s hidden states, so a single small transformer can produce drafts that agree with the target far more often than a standalone draft of the same size. The EAGLE-3 paper reports “1.4x improvement over EAGLE-2” and “speedup ratio up to 6.5x” versus vanilla (arXiv:2503.01840). The HF model card for AngelSlim/Qwen3-4B_eagle3 lists a 0.2B-parameter EAGLE-3 head trained for Qwen3-4B; the published 1.66x mean speedup with a 2.08 mean accept length is what to plan on. The catch is that the EAGLE-3 head is target-specific: an LLM that does not have a published head cannot use this strategy without retraining.
draft-dflash and draft-dspark. DFlash is a small block-diffusion draft that emits a whole block of draft tokens in one forward pass, then injects the target’s hidden states into the draft’s attention. llama.cpp’s docs note it “keeps the draft model small while making drafting GPU-friendly” (llama.cpp speculative docs). DSpark adds a semi-autoregressive Markov head on top of DFlash. Both are draft-model-based, so they have the same VRAM-overhead catch as draft-simple. At last review llama.cpp documented DSpark as “Currently only drafts with a Qwen3 backbone are supported; support for other backbones (e.g. Gemma4) is planned.”
draft-mtp. Some models ship native Multi-Token Prediction heads; if the target has them, no separate draft model is needed. vLLM’s method table flags this as “Best when the target model has native MTP support.” Gemma 4 assistant checkpoints are loaded as MTP speculators in vLLM rather than as generic draft models (vLLM speculative decoding docs).
ngram-*. The n-gram family needs no extra model at all. llama.cpp ships five variants. ngram-mod maintains a hash pool shared across requests, weighs ~16 MB, and works best on long outputs with repeated phrases (source-code refactors, reasoning models that loop back into prior text). The --spec-ngram-mod-n-min and --spec-ngram-mod-n-max ranges the docs publish: “small n are not recommended,” “MoEs require long drafts,” “dense models: can reduce --spec-ngram-mod-n-min and --spec-ngram-mod-n-max.” Acceptance is workload-specific. The docs’ example statistics output shows a 0.70 draft acceptance rate in an ngram-mod run; that is a sample printout, not a benchmark, and the speedup depends on how repetitive the prompt and output actually are.
When it backfires
Three real ones, all from the docs or from the open issues.
- Acceptance is low. If the draft and target disagree often, the verifier pays for every rejected proposal and you can end up below vanilla. llama.cpp’s synthetic-acceptance benchmark flag (
--spec-synth-rates,--spec-synth-len) lets you replace the verifier’s decisions with synthetic acceptance rates to size the speedup before you commit to a draft model. - The draft model is the wrong size or wrong family. Pairing a draft that is too small, too large, or from a different training distribution gives lower gains and sometimes negative gains. The published EAGLE-3 numbers (HF card) sit at a 0.2B head on a 4B target; a head trained for a different target does not carry those numbers over. The same trap exists for the target: if the chosen target’s quant ladder eats the VRAM the draft needs, no published speedup applies to your VRAM tier.
- The runner does not expose it. Ollama’s draft-model support exists for some configurations and not others; a closed issue thread tracked “MTP speculative decoding regression with Qwen3.8 27B” in v0.34.2 (closed September 19, 2026) and a closed “What are the Muse and DFlash plans?” question issue (closed August 13, 2026) (Ollama speculative-decoding issues). If your runner does not expose the strategy, the published number does not apply to you.
The Hugging Face Transformers generation strategies page documents a custom_generate API for loading custom decoding code from the Hub; it does not present speculative decoding as a ready option, so a draft loop there is something to write, not switch on.
How to turn it on
For llama.cpp’s llama-server, the relevant flags are --spec-type, -md / --spec-draft-model / --hf-repo-draft, --spec-draft-n-max (draft length per step, default 3), and --spec-draft-ngl for VRAM layers. For n-gram, --spec-type ngram-mod --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 is the docs’ sample usage. The full table of strategies and CLI flags is on the llama.cpp speculative decoding documentation page; the vLLM analogue takes a JSON blob to --speculative-config whose keys are spelled out on the vLLM speculative decoding documentation page.
Before turning it on, check the runner status. Ollama support is partial; llama.cpp and vLLM ship the broadest sets, and neither ships every strategy the other does.
The bottom line
Speculative decoding is real, and the published figures run from 1.4x (AngelSlim’s lower bound on vLLM) to 6.5x (the EAGLE-3 abstract’s best case), all reported by the people who built the method; on short outputs or on workloads with low draft-target agreement it can be zero or negative. EAGLE-3 reaches higher acceptance than a standalone draft of the same size, for targets that have a trained head; a plain draft model is the most-used; n-gram methods are the cheapest to try because they need no extra weights. Pick the runner that ships the strategy, pick a draft whose target matches your workload, and use the runner’s synthetic-acceptance benchmark to size the win before committing to a draft download.
The strategy name does not move generation t/s on its own. Acceptance does. Acceptance comes from the right draft on the right target, or from a workload where the text actually repeats.