I turned on MLX's memory-saving flag and ran out of memory

Jun 9, 2026

On a 16 GB Mac, MLX’s --kv-bits flag does the opposite of what it’s for. Its whole job is to shrink the KV cache — quantize the keys and values to 4 or 8 bits so longer contexts fit in memory. Instead, it raised peak memory at every context length I measured, and at 32K tokens it ran out of memory on a machine where plain 16-bit KV fit comfortably at 9.4 GB. It isn’t faster either (8-bit decoding ran ~4× slower in my tests; 4-bit lands about on par), and it doesn’t buy you any quality you’d want in exchange. On this code path, the honest advice is: leave it off.

I went looking for the opposite result. The KV cache is the thing that decides whether a long prompt fits on a small Mac — for Llama-3.1-8B it grows at ~131 KB per token, so a 32K context is ~4 GB of pure cache on top of the model. Quantizing that cache to 4 bits should cut it to a quarter. I expected a clean “kv-bits saves you 1–3 GB” post. I measured the flag instead.

The measurement

I measured peak memory under a real long-context load — one model per process (so the peak is clean, not a leftover from a previous run), prompt sized to the target context, KV quantized from the first token. The number that matters is peak, because peak is what OOMs you.

The peak numbers, Llama-3.1-8B-4bit on the M3:

Every quantized cell is higher than the fp16 cell next to it, and the penalty grows with context: +1.1 GB at 8K, +2.0 GB at 16K, and at 32K the 4-bit cache crashes with a Metal “Insufficient Memory” abort on a machine that runs the full-precision cache at 9.4 GB. The flag whose selling point is “fit a longer context” shortened the longest context I could fit.

This isn’t a Llama quirk. I re-ran the whole sweep on Qwen3-4B — a different model family — and got the same shape: 3.97 → 4.94 GB at 8K, 5.18 → 7.04 GB at 16K. Quantizing the cache cost memory on both models. The only place 4-bit and fp16 tied was at 2K tokens — where the cache is so small it doesn’t matter either way.

Why a “smaller” cache needs more memory

The stored cache really is smaller in 4-bit. But peak memory isn’t about what’s stored at rest — it’s about the largest transient allocation during a forward pass, and that’s where the quantized path loses.

Full-precision KV runs through MLX’s fused attention kernel (mx.fast.scaled_dot_product_attention), which computes attention without ever writing the full score matrix to memory — that’s the entire point of a fused, flash-attention-style kernel. Quantized KV can’t use it. It falls back to a hand-rolled path (quantized_scaled_dot_product_attention in mlx-lm’s models/base.py) that does the opposite: it explicitly builds the full query×key score matrix — shape [heads, query_len, context_len] — and softmaxes it in memory before the second matmul.

During prefill the query chunk runs up to 2,048 tokens while the context grows to the full length, so that score matrix becomes a multi-gigabyte transient that scales with context. That’s the widening gap in the chart: the 4-bit cache really is smaller at rest, but the unfused attention you need to read it allocates more than the cache ever saved — and the overage grows with context until, at 32K, it shoves a 9.4 GB workload past 16 GB and the Metal allocator gives up.

I checked this in the source rather than inferring it from the curve: the fp16 branch calls the fused kernel, the quantized branch materializes the score matrix explicitly (base.py, the hasattr(cache, "bits") split). The one piece I didn’t separately profile is how much the per-chunk cache re-quantization adds on top — but the score-matrix materialization alone accounts for the shape and the OOM.

No speed upside, and 8-bit is a trap

If kv-bits cost memory but bought speed, it might still be worth it. It doesn’t. 4-bit decoded at about the same rate as fp16. 8-bit decoded at 1.6 tokens/sec versus fp16’s 6.6 at 8K — roughly 4× slower — while using more memory. Eight-bit KV is the worst cell on every axis at once: slower than fp16 and hungrier than 4-bit. (Token-rate numbers here are noisier than the memory numbers, because my generations were short; treat the 4× cliff as the robust claim, not the second decimal.)

And no quality you’d trade for

The last possible defense: maybe quantizing the cache costs memory and speed but preserves long-context recall well enough to matter. So I planted three facts at different depths in a 12K-token prompt and asked for each back, at fp16 and at 4-bit. Both scored 3/3 — identical. Four-bit KV didn’t lose anything. Which, given everything above, is the point: there’s no quality cost, but there’s no quality benefit either. You’d be paying memory and speed for nothing.

(That recall test also caught a bug in my own grader — it marked the correct answer “4,182 copper coins” as wrong because it was grepping for the filler word “exactly” instead of the number. I’ve been keeping a running tally of bugs my benchmarks have shipped; this is another for the list. Spot-check your failures, not your passes.)

The upshot for 16 GB

On this MLX path, --kv-bits is all cost and no benefit: more memory, no faster, no better. The honest lever for fitting long context on a small Mac is the boring one — run the cache in fp16 and accept that it grows ~131 KB/token linearly, then stop before it eats your RAM. Quantizing the cache doesn’t push that wall further out; it pulls it closer.

One more thing worth knowing before you reach for this flag on a Gemma-style sliding-window model: it doesn’t quietly do nothing — it errors out. Those models build a RotatingKVCache for their sliding layers (Gemma-3-1B uses 22 of them alongside 4 full-attention layers), and RotatingKVCache.to_quantized() is unimplemented in mlx-lm, so the first quantization step raises NotImplementedError: RotatingKVCache Quantization NYI. I confirmed it: --kv-bits 4 on Gemma-3 crashes outright. KV-cache quantization simply isn’t available for that whole class of model yet.

The general lesson is the one that keeps paying out: a flag’s value lives in its implementation, not its description. “Quantize the KV cache” sounds like free memory. Measured on real hardware, under a real load, at the peak that actually OOMs you — it’s the opposite.