I built self-speculative decoding for MLX. On an M3, naive layer-skip never beats baseline — 24 configs, 24 losses

Jun 18, 2026

Self-speculative decoding — where a model drafts its own tokens by skipping some of its layers, then verifies them with the full model — does not pay off on an M3 when you pick the skipped layers naively. I built it for MLX (mlx-lm ships only the two-model kind), proved it’s lossless, and swept 24 configurations across two models. Every single one was slower than just running the model normally: the best case was 0.80× baseline speed, the worst 0.22×. The win only exists if you choose which layers to skip far more carefully than “drop the middle third” — which is exactly the problem a 2026 paper, KnapSpec, formalizes as a knapsack.

Why self-speculative decoding should work

Speculative decoding speeds up generation by having a cheap draft model guess several tokens, then letting the expensive target model verify them all in one forward pass. When the draft guesses right, you get multiple tokens for the price of one verification. mlx-lm ships this — but it needs two models (e.g. Llama-3.1-8B verified by a 1B draft), which means a second download and second chunk of your 16 GB.

Self-speculative decoding removes the second model. The draft is the target model with some layers skipped. No extra weights, no extra download. The appeal on a memory-bound Mac is obvious: you get speculative decoding’s speedup without speculative decoding’s memory tax.

Paper → the idea

The technique traces through Draft & Verify and SWIFT to KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem (arXiv:2602.20217). KnapSpec’s framing is the tell: choosing which layers to skip is not a heuristic you eyeball, it’s a constrained optimization — maximize the draft’s token-acceptance rate subject to a compute budget, with the per-layer costs depending on your hardware. The paper even justifies using cosine similarity between hidden states to predict acceptance.

That paper is the answer to a question. This post is the question: what happens if you don’t do the knapsack — if you skip a contiguous block of middle layers, the obvious first thing anyone would try?

Code → the implementation

The mechanism is small. Build the draft as a shallow copy of the model whose layer list is a subset of the same layer objects — shared weights, fewer layers, identical interface — and hand (model, draft) to mlx-lm’s existing, proven speculative_generate_step:

def build_layerskip_draft(model, skip_frac):
    n = len(model.model.layers)
    n_skip = max(1, round(n * skip_frac))
    start = (n - n_skip) // 2          # skip a contiguous MIDDLE block
    skip = set(range(start, start + n_skip))
    keep = [i for i in range(n) if i not in skip]

    draft = copy.copy(model)            # shares all weights by reference
    draft.model = copy.copy(model.model)
    draft.model.layers = [model.model.layers[i] for i in keep]
    return draft

Two details matter. First, I skip the middle and keep early and late layers — truncating the tail instead would feed the final norm and lm_head representations from a layer they were never trained to read, producing garbage drafts on a model with no early-exit training. Second, because greedy speculative decoding is mathematically lossless — the target model verifies every drafted token, so rejected guesses are discarded — the self-spec output must be identical to the baseline output. That’s a free correctness check, and it held on all 24 runs.

Hardware → the measurement

I swept skip fraction (10–50%) and draft length (2/3/5 tokens) on Qwen3-4B-4bit (36 layers) and Llama-3.1-8B-4bit (32 layers), on an M3 MacBook Air with 16 GB. Greedy decoding throughout, so every run is lossless. Speedup is self-spec tok/s ÷ baseline tok/s, measured back-to-back in the same process.

Llama-3.1-8B told the same story (best 0.70× at 10% skip). Across both models, all 24 configurations, the best result was a 20% slowdown and the typical result was 2–4× slower.

The shape is the whole point. At a small skip fraction the draft accepts most of its tokens (68%) but it ran almost the entire network to produce them — it’s nearly as expensive as the model it’s trying to accelerate, so even high acceptance can’t buy back the overhead. At a large skip fraction the draft is genuinely cheap but it has become a different, dumber model: 6% acceptance means 94% of its guesses are thrown away, and you’ve paid for them. Naive middle-skipping forces you to pick a point on a curve where both ends lose.

What it means for 16 GB users

Self-speculative decoding is not free speed on Apple Silicon. The version you’d build in an afternoon — skip the middle, run it — costs you throughput at every setting I tried, even though it’s perfectly lossless and adds zero memory. The reason it loses is structural, and it’s the same memory-bandwidth story that governs everything on these machines: a draft that skips few layers isn’t cheap enough, and a draft that skips many isn’t accurate enough, so the acceptance-rate-times-draft-cost product never clears the bar.

That is precisely the gap KnapSpec closes by making layer selection a hardware-aware knapsack instead of a guess. The honest takeaway isn’t “self-spec doesn’t work on a Mac” — it’s “self-spec on a Mac lives or dies on which layers you skip, and naive selection reliably dies.” If you want the win, you have to solve the optimization the paper describes; you cannot eyeball it.

Note: absolute tok/s drifts between sessions on a passively-cooled M3, so the trustworthy figure is the self-spec-vs-baseline ratio measured back-to-back within one run — which is how every speedup above was computed. The lossless check and the “no config exceeds 1.0×” result reproduced on re-run.