Carmack's right about the weights. The KV cache is the part his argument skips.

Jul 12, 2026

John Carmack’s pitch for running LLM inference off flash instead of HBM works because weights are read the same way every single time. The KV cache breaks that assumption in three ways at once — it grows, it’s rewritten every token, and I have the M3 numbers showing exactly how badly that combination fails the moment you try to bound it.

What Carmack actually said

On July 6, 2026, Carmack argued that AI accelerators are memory-constrained for a fixable reason: “model inference can have a deterministic memory access pattern… you don’t need ‘random access memory’ at all for model weights, and you could tolerate cold-start latencies in the multiple milliseconds, as long as continuous reads were delivered at the necessary bandwidth.” NAND flash is over 100× cheaper per GB than HBM. If you know exactly which bytes you’ll read and in what order, you don’t need RAM’s instant-access guarantee — you need a fat pipe and a good prefetcher. He flags one weak spot himself: flash wears out from writes, which is why he says the case is stronger for inference than training.

That’s a clean argument, and it’s correct for what it’s actually about: weights. Every forward pass reads the same tensors in the same order. Nothing about that changes from token to token, or request to request. It’s the textbook case for streaming from a cheap, sequential-friendly medium.

The KV cache is the thing his own caveat should have caught

Carmack’s write-durability caveat stops at training vs. inference. But there’s a third case sitting inside inference itself that behaves like training’s worst case, every single token: the KV cache.

It fails the “deterministic access pattern” premise in three ways weights don’t:

It grows. Weights are a fixed size for the life of the model. The KV cache starts small and gets bigger with every token you generate, for as long as the conversation runs. I measured this directly on an M3 running Llama-3.1-8B: peak memory climbed from 5.3 GB at a 2k-token prompt to 9.3 GB at 32k — nearly double, and still climbing, because there’s no natural ceiling short of your hardware’s limit.

It’s written every token, not just read. Weights are read-only during inference — that’s exactly why Carmack’s pitch works for them. The KV cache is the opposite: every new token appends a fresh key and value. That’s the same write-wear problem Carmack already flagged for training, except it’s happening during plain inference, on every request, forever.

Its read pattern isn’t fixed either. Attention reads across the entire accumulated cache at every step — a different, growing range each time, not the same static tensor walk you get with weights.

What happens when you try to force it to behave like weights anyway

The obvious fix is the one every serving stack already reaches for: cap the KV cache at a fixed size — a rotating window — so it stops growing and starts looking like the fixed, bounded thing flash streaming wants. I configured a cap of 4096 tokens and measured exactly what that costs, on the same M3, same model. One methodology note first: MLX’s rotating cache enforces that cap exactly during decode, but during prefill — the initial pass over a long prompt, processed in chunks rather than token by token — it can transiently hold more before trimming back down. With mlx-lm’s default chunking, that transient peak works out to roughly 6,100 tokens, not a hard 4,096. The cap is still what makes the memory story below true — a fixed ceiling regardless of conversation length — it just sits a bit higher than the configured number alone suggests.

Capping the cache does exactly what you’d want on the memory side — it stops growing, plateauing at 6 GB instead of climbing to 9.3. But the instant the conversation outgrows the window, recall doesn’t degrade gracefully. It collapses completely: 5 out of 5 planted facts down to 0, on every trial, no partial credit. The facts I’d planted were near the start of the document; a rotating window keeps only the most recent tokens, so the moment the window fills, the earliest content — exactly where the facts lived — is the first thing evicted.

I also tested the standard fix for that: pin a handful of “attention sink” tokens at the very start so eviction doesn’t wreck the model’s output. It didn’t help, because it isn’t built to help with this. Attention sinks exist to keep the softmax from destabilizing when you evict aggressively — to stop the model from degenerating into repetition or garbage. They were never designed to preserve arbitrary important content that happens to live in the middle of a long document, and my numbers show that gap directly: the sink tokens kept output coherent, and the model still forgot every fact.

Why this matters more than a rebuttal-tweet’s worth

None of this makes Carmack wrong. It sharpens where the idea actually applies. Streaming weights from flash is a genuinely good idea because weights are the one part of inference that behaves exactly the way flash wants: fixed, read-only, identical every pass. The KV cache is the part of the same system that does the opposite of all three of those things, and bounding it to make it flash-friendly doesn’t just cost you memory bandwidth — it costs you the thing the cache exists to provide in the first place.

If flash-streamed weights become how inference actually gets built — and the economics genuinely argue for it — the KV cache doesn’t get to ride along for free. It needs its own answer, and “shrink it until it fits the same box as the weights” is a real, measured way to lose the conversation’s memory, not a free trick.

Data and harness: github.com/robertlangdonn/kv-cache-tax (the growth curve is v1; the eviction table is the newer leg, same repo).