Three ways to make an LLM read its weights less often on a Mac — and why each one backfires

Jun 14, 2026

On a Mac, the slow part of generating text isn’t the arithmetic — it’s reading the model’s weights out of memory, once per token. So the obvious way to go faster is to read the weights less often. I measured three different techniques that all do exactly that on a 16 GB M3, and every one of them backfired in a different way. Speculative decoding helps a little, then tips negative if you push it. Diffusion generation ties the best model on quality but is slower, not faster. Self-speculative layer-skipping — which I had to build from scratch because nothing in the MLX ecosystem does it — never beats plain decoding at all. The common thread is the interesting part: each trick moves the bottleneck off memory bandwidth and onto something else, and on this hardware the something else is worse.

Why “read the weights less often” is the whole game

Generating one token, autoregressively, means a full forward pass through the model: every weight gets read from unified memory exactly once. On an M3, that read is the bottleneck — the compute units finish their math and sit idle waiting for the next slab of weights to arrive. Decode throughput is, to a first approximation, memory bandwidth ÷ bytes-read-per-token. (This is why a 4-bit model is ~2× faster than the same model at 8-bit: half the bytes per read.)

Every technique below attacks the number of reads. If you can produce N tokens while reading the weights fewer than N times, you should win. All three deliver on that arithmetic. None of them win cleanly. Here’s each, measured.

Trick 1: speculative decoding — a 20% win that becomes a 25% loss

Speculative decoding runs a small, cheap “draft” model to guess several tokens ahead, then verifies all of them with the big model in a single forward pass. When the guesses are right, you got several tokens for one expensive read of the big model’s weights.

Measured on an M3 — a 1B draft against an 8B target — it works, modestly: +20% at num_draft_tokens=2 (7.99 → 9.6 tok/s). But the knob is sharp. At num_draft_tokens=4 it goes −25%, slower than no draft at all. Each rejected guess is wasted draft work plus a wasted slot in the verify pass, and on a bandwidth-bound machine the verification doesn’t have spare throughput to absorb the misses. The datacenter-GPU papers report 2–3× from this; on an M3 the ceiling is much lower and the downside is one knob-click away. (full writeup)

Trick 2: diffusion — many tokens per pass, and slower anyway

A diffusion language model doesn’t generate left to right. It denoises a whole block of tokens at once — many tokens committed per forward pass, which is the most direct possible attack on weight-reads-per-token. I ran LLaDA2-mini on the M3 expecting the speed win this should obviously produce.

It tied my best model on quality (20/21, level with Qwen3-4B) — genuinely surprising for a different architecture. But it was not faster: its decode speed swung ~30× by prompt (0.6 tok/s on a two-word answer, 17.6 on a long one) and never beat the fastest autoregressive model. It commits ~4.4 tokens per pass on a long answer — but each pass processes the entire canvas of a 16B model, so the trips it saves cost more than the trips it makes. It moves work off the bandwidth axis and onto the compute axis, which on an M3 is the wrong direction. (full writeup)

Trick 3: self-speculative layer-skipping — never wins

The third idea drops the separate draft model: let the big model draft for itself by running only a subset of its own layers, then verify with all of them. Training-free, no second model to load. Nothing in the MLX ecosystem does this — mlx-lm only has two-model speculative decoding — so I built it: a draft that’s the same model with a block of middle layers skipped (keeping the early and late layers, so the output head still sees sensible representations), sharing weights with the full model.

It’s correct — greedy self-speculation is lossless, and the output matched plain decoding on every run. But across 24 configurations (two models, four skip fractions, three draft lengths) not one beat baseline. Best case: 0.8× — still 20% slower. The two failure modes squeeze with no gap between them: skip a few layers and acceptance is high but the draft isn’t cheap enough; skip many and the draft is cheap but acceptance collapses (6% at half the layers). There’s no middle where the self-draft is both cheap and faithful — which is exactly the gap that smarter layer-selection methods (like the knapsack scheme in the KnapSpec paper) exist to close. Naive skipping isn’t enough on this hardware.

The pattern: the bottleneck keeps moving

Three techniques, one shape. Each one really does reduce weight-reads per token — the theory holds every time. And each one relocates the cost somewhere the M3 handles worse:

• Speculative decoding moves it onto wasted verification when drafts miss.
• Diffusion moves it onto compute per pass — a 16B canvas per step.
• Self-speculative skipping moves it onto the draft-cheapness vs acceptance tradeoff, with no sweet spot.

This is the thing I keep seeing on constrained hardware: you peel back the obvious bottleneck and a new one is waiting. Memory bandwidth is the wall, so you cut reads — and the cost reappears as compute, or wasted work, or a tuning knife-edge. The honest wins on a 16 GB Mac are still the boring structural ones: a smaller model, a mixture-of-experts with few active parameters, 4-bit weights. The clever pass-reduction tricks need either careful tuning (speculative decoding at exactly num_draft=2) or smarter machinery than the naive version (KnapSpec-style layer selection) before the theory turns into wall-clock.

That’s not a knock on the techniques — it’s where the frontier is. Every one of these has a regime where it wins; that regime just usually isn’t a single stream on a bandwidth-bound laptop. Knowing which bottleneck you’ve moved the work onto is the whole skill, and on an M3 the answer is rarely “and now it’s free.”