Speculative decoding on a 16 GB Mac: a 20% win that becomes a 25% loss
Speculative decoding gives a real but modest speedup on an M3 — about 20% — and the knob that controls it is sharp enough to turn that win into a 25% slowdown. On a 16 GB MacBook Air, a 1B draft model accelerating Llama-3.1-8B peaks at num_draft_tokens=2; push it to 4 and you decode slower than with no draft at all. That’s what over-drafting looks like on a machine whose decode is bound by memory bandwidth — and a reminder that inference settings don’t always transfer from a GPU to a laptop.
Two draft tokens is the sweet spot (+20%); the gain decays by three and goes negative at four — slower than no draft at all. Past the sweet spot, each extra draft token is another draft-model pass whose output gets rejected before it’s used, so the curve turns down fast.
Speculative decoding is the trick where a small, fast “draft” model proposes the next few tokens and the large model verifies them in a single forward pass. When the guesses are right, you get the big model’s quality at closer to the small model’s speed. On datacenter GPUs it’s often a 2–3× win, so it’s tempting to reach for --draft-model on a laptop too.
I measured it on an M3: Llama-3.1-8B-4bit as the target, Llama-3.2-1B-4bit as the draft. The setting that matters is num_draft_tokens — how many tokens the draft guesses before the target checks them.
| num_draft_tokens | decode tok/s | vs baseline |
|---|---|---|
| none (baseline) | 8.0 | — |
| 2 | 9.6 | +20% |
| 3 | 8.8 | +10% |
| 4 | 6.0 | −25% |
The win is real at 2, already fading at 3, and underwater at 4. It also costs ~0.7 GB of RAM to keep the draft model resident — not free on a 16 GB machine.
Why the cliff
Speculative decoding works because the target model runs once per step no matter how many tokens it checks. MLX runs the draft model num_draft times to propose tokens, then the target verifies all of them in a single forward pass — so the target’s expensive weight load is amortized across several tokens instead of one. That’s the win.
The cost grows differently. Each extra draft token is another full, sequential pass over the draft model, and acceptance falls off fast: verification stops at the first token the target disagrees with, so a deep draft token only counts if every token before it already did. Push num_draft too high and you’re running draft passes whose tokens get rejected before they’re used. Past a couple of tokens, that growing draft overhead outweighs the shrinking number of tokens you actually keep, and throughput drops below baseline.
Decode is memory-bandwidth-bound throughout — you stream the model’s weights for every pass — so those extra draft passes aren’t free; they only pay off when the guesses stick. That’s the whole game on a laptop: draft just enough to amortize the target, not so much that the draft work dominates.
Practical takeaway: if you use --draft-model on Apple Silicon, set num_draft_tokens=2 and leave it there. Higher values — the kind you’ll see in GPU-oriented setups — can, on a 16 GB Mac, make you slower than no draft at all.
One honest caveat: I used a deliberately predictable prompt, which favors high draft acceptance — so +20% is closer to a best case than a typical one, and on less predictable text the negative cliff arrives sooner. The direction is robust even if your mileage on the magnitude varies.