A rotating KV cache saves 36% of your memory and 100% of your recall

Jul 12, 2026

Capping a model’s KV cache instead of letting it grow without bound saves 36% of your peak memory at 32k context. It also drops recall of five facts I’d planted in the prompt from 5 out of 5 to 0 out of 5, the instant the conversation outgrows the cap. Not gradually. Not “loses one or two.” All five, every single trial, eight recorded runs with zero exceptions — eleven if you count the warmup passes too.

This is the follow-up to the KV-cache tax post — same model, same laptop, same four context lengths. That post asked what long context costs. This one asks what you get back if you refuse to pay the full price.

The setup

Same Llama-3.1-8B, 4-bit, same M3 16 GB. Same growing document with five “access codes” and the same end-of-prompt question. The only change: instead of the default cache, which remembers every token, I built one that caps out at 4096 tokens — a RotatingKVCache, the same mechanism behind “sliding window attention” in a lot of production serving stacks. I also kept the first 4 tokens pinned regardless of window position, since that’s the standard trick (called an “attention sink”) for keeping eviction from making the model’s output degenerate into garbage.

One methodology note worth stating up front: MLX’s rotating cache enforces that 4096-token cap exactly during decode, one token at a time. During prefill — the initial pass that processes a long prompt in chunks rather than token by token — it can transiently hold more than the cap before trimming back down, since a whole chunk lands before the trim happens. With mlx-lm’s default prefill chunk size (2048 tokens), that transient peak works out to roughly 6,100 tokens’ worth of cache, not a hard 4,096. The cap is still what makes the memory story below true — it’s a fixed ceiling regardless of how long the conversation gets — the ceiling itself just sits a bit higher than the configured number alone would suggest.

The memory story is exactly what you’d hope

Peak memory stops growing. Once context passes the window, the cache simply can’t get any bigger — old tokens fall off the back as new ones come in. At 32k that’s the difference between 9.3 GB and 6.0 GB, more than a third of the memory bill, gone. If all you cared about was “does it fit,” a rotating window is a clean win.

The recall story is not a curve, it’s a cliff

I expected some degradation — maybe the model starts losing the earliest fact first, then the next, a gentle slide as the window eats further into the document. That’s not what happened. Recall is 5/5 right up until the window is crossed, then it’s 0/5, and it stays 0/5 all the way out to 32k. There’s no partial credit. The facts are either in the window or they’re gone, and once the conversation has grown past it, they’re gone.

That makes sense once you think about why: a rotating cache keeps the most recent tokens. My facts are near the front of the document. The moment the document outgrows the window, the front — where the facts live — is exactly the part that’s been evicted. This isn’t a subtle failure mode. It’s the window doing precisely what it’s built to do, applied to content that happened to be in the wrong place.

The sink tokens don’t help, and that’s the actual finding

Here’s the part I didn’t expect going in. I kept the first 4 tokens pinned — the standard “attention sink” setup, the same default MLX’s own RotatingKVCache ships with. Attention sinks exist because dropping the very first tokens of a sequence can make eviction-based generation degenerate into repetition or garbage; keeping a small anchor at the start fixes that. It’s a real mechanism and it works for what it’s for.

It did nothing for recall here, because it was never going to. The sink protects positions 0 through 3. My facts aren’t at positions 0 through 3 — they’re scattered through the body of a document that’s thousands of tokens long. Attention sinks keep a model’s output coherent under eviction. They don’t keep the model’s memory of arbitrary content intact. Those are two different jobs, and it’s easy to read “keeps the model from breaking” as “keeps the model from forgetting” when they’re not the same claim. This data is what that difference looks like when you measure it instead of assuming it.

Why I care about this

Every serving stack that offers a memory-saving KV policy — sliding windows, StreamingLLM-style attention sinks, sequence-length caps — is making the same trade this experiment makes explicit: memory for retention. The sink-token default makes the trade look safer than it is, because it solves the visible failure mode (garbled output) and leaves the invisible one (silent factual amnesia) completely unaddressed. If you’re running long-context agents on constrained hardware and reaching for a rotating cache to make things fit, this is the cost you’re not being warned about by the fact that generation still looks fluent.

Caveats, read before you quote this

• Early-needle test, not scattered. The five facts land near the front of the document, same caveat as the v1 post. A rotating window’s recency bias means this is close to a worst case for where the facts could be — the finding would look different (though probably not better) with facts placed throughout or near the end.
• One window size, one keep value. 4096 and 4 are the mlx-lm/StreamingLLM defaults, not a sweep. Whether a larger window or bigger sink changes the cliff shape is the natural next question, not yet answered here.
• n=3 per point except 32k, which is n=2. The machine hit real memory pressure (swap over 90% of an 8 GB file) partway through the third 32k pass, twice, and I killed the run both times rather than push through and risk degrading the rest of the system. Both recorded 32k trials agree with each other and with all eight recorded above-window trials combined — I’m confident in the result, but the methodology gap is real and I’m not hiding it.
• M3 only, this time. vLLM’s sliding-window support is tied to models architecturally trained with it (Mistral-shaped), not a generic runtime cap you can bolt onto any checkpoint the way mlx-lm’s cache classes let you. A clean cross-hardware version of this leg needs more design work than a rental round, so it’s not attempted yet.

Code and raw data: github.com/robertlangdonn/kv-cache-tax — see the “Eviction leg” section of the README.