3 posts tagged “benchmark”
Apple's on-device model ties a 4-bit Llama-3.1-8B — and won't name the M1
Apple shipped an official Python SDK for its on-device Foundation Model at WWDC 2026. I put the ~3B model through the same 21-task quality suite I use for MLX models: it ties a 4-bit Llama-3.1-8B (18/21), one question behind Qwen3-4B. Quality is the only fair axis to compare — and that limitation is itself the interesting part.
One flag makes Qwen3-4B beat Llama-3.1-8B on a 16 GB Mac — at half the RAM
On an M3 MacBook Air, Qwen3-4B with the thinking trace turned off scores 20/21 on a verifiable suite — beating Llama-3.1-8B's 18/21 at half the memory and nearly double the speed. With thinking on, the same model drops to 7/21. The flag is enable_thinking=False, and here's exactly what it changes and why it matters.
Gemma 4 12B on a 16 GB Mac: 11 GB RAM, 2.7 tok/s, and what my benchmark got wrong
Google's Gemma 4 12B uses 11.4 GB of RAM and runs at 2.7 tok/s on an M3 MacBook Air — 2.4× the memory of Llama-3.1-8B at well under half the speed. Its math and factual answers are flawless; its coding can't be cleanly scored. Here's the honest picture, the multimodal tax, and the benchmark bug I found correcting this post.