6 posts tagged “benchmarks”
Atomic Chat's TurboQuant headline did not survive a chat-generation benchmark on my M3
Atomic Chat advertises TurboQuant as 8x faster inference and 6x less memory. I tested the local MLX TurboQuant KV path on a 16 GB M3. It saved about 3-5% total peak memory and did not speed up generation — a useful reminder that KV-cache microbenchmarks do not automatically become whole-chat product claims.
I built self-speculative decoding for MLX. On an M3, naive layer-skip never beats baseline — 24 configs, 24 losses
Self-speculative decoding lets a model draft its own tokens by skipping layers — speculative decoding's speedup with no extra memory. I built it for MLX and swept 24 configs on an M3. Every one was slower than baseline, even though all were lossless. Here's why, and the paper that fixes it.
I expected a diffusion LLM to be fast on my Mac. It tied the best model on quality instead — and lost on speed.
LLaDA2.0-mini, a diffusion language model, runs on a 16 GB M3 and ties Qwen3-4B for the best answer-quality score I've measured (20/21). But it's slower than the fastest autoregressive model and uses 4× the memory of the lightest — and the exact reason I expected it to be fast on bandwidth-bound hardware turned out to be why it isn't. A measured look at where the bottleneck actually moved.
My benchmark graded '7! = 5040' as wrong — and two other ways it lied to me
Re-running my own LLM benchmark, I found a bug that had inflated the quality scores in posts I'd already published. Then a second bug. Then a third. Here's how a wrong number looks exactly like a right one — and why you spot-check the failures, not the passes.
Gemma-3-12B QAT vs Qwen3-14B 3-bit: same quality on a 16 GB Mac, but the smaller model runs lighter and faster
Benchmarking Gemma-3-12B, Qwen3-14B, and Llama-3.1-8B on a 16 GB MacBook Air (M3) with MLX. A quantization-aware 3-bit 12B ties a naïve 3-bit 14B on overall answer quality while running faster and in less memory. On a memory-bound Mac, a well-quantized smaller model can match a bigger naïvely-quantized one — so parameter count alone is the wrong thing to shop on.
What actually runs well on a 16 GB MacBook
Honest local-LLM benchmarks on a base M3, 16 GB — tokens/sec, peak RAM, and exactly where it hits the wall. The numbers nobody publishes because they run on H100s.