KV Cache Quantization Is 4× Slower on My Mac and 28% Faster on a Rented L4

Jul 19, 2026

Quantizing the KV cache made the same class of long-context inference knob about 4x slower on my MacBook and up to 28.4% faster on a rented NVIDIA L4. The idea was the same in both cases: store attention keys and values in fewer bits so decode has fewer bytes to read. The outcome was not the same. On MLX on an M3, --kv-bits raised peak RAM, slowed decode, and failed at 32k where fp16 KV fit. On vLLM 0.25.1 on an L4, fp8 KV doubled the token capacity at the same reservation and got faster as context grew. KV cache quantization is not one behavior. It is an implementation and hardware behavior.

The Mac result was the surprising one first. In June 2026, I measured MLX’s --kv-bits path on an M3 and found the opposite of the promised shape. Peak RAM went up at every context length. 8-bit KV decoded at about 1.6 tok/s versus 6.6 tok/s for fp16 KV, roughly a 4x slowdown. The quantized-KV run also OOM’d at 32k where fp16 KV still fit.

I wrote that up as the memory-saving flag that backfires. At the time, I also checked the mechanism in mlx_lm rather than stopping at the benchmark table. The quantized SDPA path materialized the full attention score matrix, shaped like [heads, L, ctx]. The fp16 path used a fused kernel that never had to build that matrix. So the “smaller KV cache” path was smaller in storage format but larger in the actual attention implementation. That is how a memory-saving flag can raise peak memory. Very rude, but mechanically unsurprising once you look at the code.

The L4 result has the shape I originally expected from KV quantization. I rented an L4, used vLLM 0.25.1, ran Llama-3.1-8B AWQ-INT4, and compared fp16 KV against fp8 KV with the same 5-planted-fact transcript, grown from 2k to 32k, that the KV-cache tax harness uses. Both arms ran in the same rental session. The reported numbers are medians of 3 interleaved passes.

The capacity result was cleaner than the speed result. At the identical 13.37 GiB KV reservation, fp16 KV gave 109,504 tokens and fp8 KV gave 219,008 tokens. That is exactly 2.0x. Time to first token improved slightly at 32k too, from 14.79 seconds to 14.11 seconds. I would not build a thesis around that TTFT delta, but it did not move in the wrong direction.

The mechanism is simple enough. Batch-1 decode spends a lot of time reading memory. For every generated token, the runtime needs to stream the model weights and the relevant KV cache through the GPU. The weights are mostly a fixed cost for this comparison. The KV cache grows with context. If the attention kernel can keep the KV cache in fp8 and dequantize it inline without adding the memory traffic back somewhere else, decode should benefit more at longer contexts. That is what the L4 curve shows.

This is also why the Mac and L4 results do not contradict each other. On the L4, vLLM’s fp8 KV path behaves like a first-class long-context optimization. On the M3 result I measured in June, MLX’s quantized path behaved more like a compatibility path that lost the fused attention behavior. One path reduces the bytes that matter during decode. The other path creates a larger intermediate and pays for it.

That distinction matters when reading benchmark posts, including mine. “Does KV cache quantization help?” is not a well-formed question. The answer needs the runtime, hardware, model, batch size, context range, kernel path, and quality bar. Otherwise the claim is too vague to reproduce and too easy to overgeneralize. It is the same trap I measured in Atomic Chat’s TurboQuant claim: a KV-cache microbenchmark that did not survive as a whole-chat number.

The practical version is narrower. On vLLM 0.25.1 on an L4 at batch 1, with Llama-3.1-8B AWQ-INT4, fp8 KV doubled the KV capacity and improved decode by up to 28.4% at 32k on my harness, with 5/5 planted-fact recall in both arms. On MLX on an M3 in June 2026, --kv-bits was worse than leaving KV in fp16 for my long-context test. Both statements can be true. Neither should be copied onto a different stack without measuring.

I also replicated the same general shape on Qwen3-4B. Comparing bf16 KV against fp8 KV, capacity went from 83,488 tokens to 166,992 tokens, again 2.0x. Decode improved from about +1% at short context to +21% at 32k. That replication does not make fp8 KV universally safe. It does make the L4 result look less like a one-model accident.

For Mac users, my advice from the June MLX post has not changed. If you are trying to stretch long context on a 16 GB Mac, do not assume --kv-bits will save you. On the version I measured, fp16 KV gave the better memory and speed behavior. Linear RAM growth is boring, but boring is useful when the alternative is a slower path that OOMs earlier.

For people self-hosting on NVIDIA with vLLM, fp8 KV is worth testing — in my cost audit of the same model on this same GPU, FP8 weights plus FP8 KV cut cost per million output tokens by 64%. It is one flag, and on this L4 run it bought back more than a quarter of decode time at 32k while doubling KV capacity. I would still put it behind your own quality evaluation before using it for production traffic. Smaller cache entries are only useful if the answers you care about survive.

Caveats

vLLM’s own startup log warns that fp8 KV may cause accuracy loss without a proper scaling factor. This run used uncalibrated dynamic scaling. My recall probe was deliberately coarse: five planted facts, retrieved verbatim. That catches a certain class of long-context failure, but it does not measure perplexity, subtle degradation, tool-use reliability, or long reasoning chains.

The two stacks also differ in weight quantization. The MLX run used MLX 4-bit weights. The L4 run used AWQ-INT4 for Llama-3.1-8B. Read each comparison within its own stack: fp16 KV versus quantized KV on the same setup. Do not compare absolute tok/s across the Mac and the L4 and pretend that isolates the KV cache.

The L4 numbers are batch 1, prefix caching off, token-counted from vLLM’s own usage field, and measured on an actively cooled rented GPU. The three interleaved passes agreed closely. The M3 thermal story is separate and less flattering to long sustained runs, which is another way laptops remind you they are laptops.

MLX also moves fast. The M3 measurement is from June 2026. The specific fused-versus-unfused gap could close in a later release. If it does, that would be a good outcome, not a contradiction. The broader point remains: KV cache quantization is not an abstract optimization you can reason about from the bit width alone. The kernel path decides whether the saved bytes stay saved.

The harness, raw data, and exact rental runbook are public here: github.com/robertlangdonn/kv-cache-tax. The L4 cost was about ₹41/hr, about 35 minutes of billed compute, roughly ₹25 total.