7 posts tagged “kv-cache”
KV Cache Quantization Is 4× Slower on My Mac and 28% Faster on a Rented L4
The same idea — store attention keys and values in fewer bits — measured on two stacks: MLX's --kv-bits on an M3 raises RAM and slows decode 4×, while vLLM's fp8 KV cache on an L4 doubles capacity and speeds decode up to 28%. KV quantization is a property of the implementation and hardware, not the technique.
Carmack's right about the weights. The KV cache is the part his argument skips.
John Carmack argued AI inference should stream model weights from cheap flash instead of expensive HBM, since weight access is deterministic. That's correct for weights — but the KV cache grows, gets rewritten every token, and reads a shifting range. I measured what happens when you force it to behave like a fixed resource anyway.
A rotating KV cache saves 36% of your memory and 100% of your recall
Capping a model's KV cache instead of letting it grow without bound plateaus peak memory — but the instant a conversation outgrows the cap, recall of five planted facts collapses from 5/5 to 0/5, every trial, with zero partial credit. The attention-sink trick doesn't help, and here's why.
At 32,000 tokens, the costliest thing my MacBook did was wait seven minutes to speak
I ran the same long-context test on a 16 GB fanless M3 and a ₹23 rented NVIDIA L4. The laptop fits a 32k context on an 8B model and keeps every planted fact — but prefill balloons to seven minutes and its decode speed can't even be measured, because the fanless chip throttles. A measured, cross-hardware look at the KV-cache tax.
Atomic Chat's TurboQuant headline did not survive a chat-generation benchmark on my M3
Atomic Chat advertises TurboQuant as 8x faster inference and 6x less memory. I tested the local MLX TurboQuant KV path on a 16 GB M3. It saved about 3-5% total peak memory and did not speed up generation — a useful reminder that KV-cache microbenchmarks do not automatically become whole-chat product claims.
I turned on MLX's memory-saving flag and ran out of memory
On a 16 GB Mac, MLX's --kv-bits flag — whose entire job is to shrink the KV cache so longer contexts fit — raised peak memory at every context length I tested, and OOM'd at 32K where plain fp16 fit at 9.4 GB. It's also no faster (8-bit decoding ran ~4× slower in my tests) and costs no quality you'd want to keep. Here's the measurement, the code-level cause, and why the flag backfires on this path.
Attention sinks: the four tokens that stabilize infinite context on a 16 GB Mac
StreamingLLM (2023) found that keeping four specific tokens in the KV cache prevents catastrophic perplexity collapse at long contexts. mlx-lm implements this as RotatingKVCache(keep=4). Here's what that means, why it works, and what our measurements on M3 actually show.