Atomic Chat's TurboQuant headline did not survive a chat-generation benchmark on my M3
Atomic Chat’s TurboQuant headline is easy to overread. The public claim says “8x faster inference” and “6x less memory,” but the fine print points at H100 attention computation and KV-cache compression. I tested the local question a Mac user actually cares about: does the TurboQuant KV path make a real chat generation faster or meaningfully smaller on an M3 with 16 GB of memory? On my machine, it did not.
I used the closest local MLX path I could test: mlx_vlm with TurboQuant 3.5-bit KV cache, running mlx-community/Qwen3.5-4B-4bit. Each cache mode ran in a fresh process so MLX peak memory could not leak across runs, and each was warmed up once at the real prompt shape — so Metal kernel-compilation cost is excluded — then measured three times. I report the median. The prompt was a repeated long-context passage about memory bottlenecks, then a short summarization request.
TurboQuant did compress the KV path, but total process memory barely moved because the model weights still dominate this setup. The speed story depends on context length: TurboQuant was 2x slower at 8k tokens and roughly at parity by 16k — never faster.
| Prompt tokens | KV mode | Peak memory | Decode speed |
|---|---|---|---|
| 8,076 | none | 5.618 GB | 25.48 tok/s |
| 8,076 | TurboQuant 3.5 | 5.452 GB | 11.69 tok/s |
| 16,128 | none | 6.379 GB | 9.14 tok/s |
| 16,128 | TurboQuant 3.5 | 6.090 GB | 8.90 tok/s |
That is a 3.0% total-memory reduction at about 8k prompt tokens and a 4.5% reduction at about 16k. Decode was 2x slower at 8k (0.46x baseline) but roughly at parity by 16k (0.97x) — the penalty shrinks as context grows and the KV cache becomes a larger share of the work. Either way, it never got faster. (Numbers are warmed-up medians of three runs per cell; absolute throughput drifts a little between sessions on a passively cooled M3, so the trustworthy comparison is TurboQuant against the baseline measured back-to-back in the same session.)
This does not mean TurboQuant is fake. It means a true KV-cache microbenchmark can become a misleading whole-product headline. On this workload, the model weights and implementation overhead still dominate enough that compressing the KV cache does not become “6x less memory” at the process level, and the extra cache work does not become “8x faster” generation.
The honest version is narrower and more useful: TurboQuant is real cache compression, but in this M3 local-chat path it saved about 5% peak memory and did not speed up generation.
That distinction matters. The hard part in local inference is not finding impressive numbers; it is preserving the boundary around what those numbers actually measure. I help teams do exactly this kind of claims audit on real hardware before a benchmark line turns into a product promise.
Evidence:
- Raw results (warmed, median of 3):
ondevice-bench/results/atomic_turboquant_bench_v2.jsonl - Finding note:
ondevice-bench/results/atomic_turboquant_finding.md - Claim source: atomic.chat
- TurboQuant paper: arXiv:2504.19874