Gemma-3-12B QAT vs Qwen3-14B 3-bit: same quality on a 16 GB Mac, but the smaller model runs lighter and faster
Update (2026-06-06). This post originally drew its quality conclusion from two prompts and called the Qwen3-14B coding result a “failure.” I later ran both models through a fuller 21-task verifiable suite (with a fixed, uniform methodology). The richer result is more nuanced and is folded in below: the two models tie on overall quality (17/21 each) — Gemma-3-12B QAT wins coding and speed, the naïve-quant Qwen3-14B wins math and factual recall. The original framing (“quantization method over size”) over-claimed — this is one matchup with several variables differing (family, size, model type), so the post now leads with what it actually showed: a tie on quality, with the smaller QAT model lighter and faster. The “QAT clearly wins on quality” framing was also too strong — it’s a tie.
Earlier I measured what runs on a 16 GB MacBook Air and found that models above ~8B tip into swap. The natural fix is to quantize a bigger model down so it fits: a 14B is ~8 GB at 4-bit (too big) but ~6 GB at 3-bit. I tested whether that works. It does, but the more useful finding was about how you quantize, not how large a model you can squeeze in.
The 3-bit models fit
Three models, same machine (MacBook Air 15″, M3, 16 GB), MLX, with my normal apps open (a real working session, not a closed-everything lab setup):
Three models on a 16 GB MacBook Air, single-stream generation speed. The smaller QAT-quantized 12B runs faster than the naïve-quant 14B and fits in less memory — the quality picture (below) is a closer, more interesting tie than this speed chart alone suggests.
All three ran, including the 14B at 4.4 tokens/sec. So quantizing down does get a bigger model over the memory wall. The real question is whether the answers hold up at 3-bit.
Answer quality
Same two prompts for each model, with enough token budget to finish: a reasoning question and a small coding task.
Reasoning (bat and ball, answer $0.05):
- Gemma-3-12B (QAT 3-bit) — correct, with clean algebra.
- Qwen3-14B (3-bit) — correct, and it self-verified. (It is a reasoning model, so it needs room to think; with a tight token budget you would wrongly conclude it failed.)
- Llama-3.1-8B (4-bit) — right answer, but a contradictory explanation.
Coding (iterative fib):
- Gemma-12B-QAT and Llama-8B — correct.
- Qwen3-14B-3bit — failed this prompt: it repeated “You need to make sure the function…” and ran out of budget before writing code. Degenerate repetition is a known symptom of aggressive low-bit quantization. (On the fuller 21-task suite it isn’t a blanket failure — it solved 4 of 8 coding tasks; the thinking-trace overhead eats into a tight budget, but it’s not broken.)
Same quality — but smaller, lighter, faster
On the fuller suite, the two 3-bit 12–14B models tie at 17/21 — but they get there differently. Gemma-3-12B QAT wins coding (6/8 vs 4/8) and runs faster, in less memory (6.1 GB vs 6.6 GB); the naïve-quant Qwen3-14B wins math (8/8 vs 7/8) and factual recall (5/5 vs 4/5). So QAT doesn’t “win on quality” outright — but it delivers equal overall quality while being faster and lighter, which on a memory-bound 16 GB Mac is the axis that matters.
The difference is how each model reaches 3-bit:
- Naïve 3-bit (Qwen3-14B-3bit): a finished model is quantized after training. Cheaper to produce, and quality drops.
- QAT 3-bit (Gemma-3-12B): the model is trained to run at low precision, so quality holds up.
So “pick the largest model that fits at 3-bit” is the wrong heuristic. For a given memory budget, a smaller QAT-quantized model can outperform a larger naïvely-quantized one.
One honest limit: this is a single matchup, and the two models differ in more than quantization — family (Gemma vs Qwen), size (12B vs 14B), and type (instruct vs reasoning). So it shows quantization quality can outweigh raw size on a fixed memory budget — not that it always does. Isolating QAT itself would need the same model in both QAT and naïve 3-bit.
What to run on a 16 GB Mac
- For speed and reliability: Llama-3.1-8B (4-bit), ~7.7 t/s.
- For the best answers that still fit: Gemma-3-12B QAT (3-bit), ~5.4 t/s — equal overall quality to the 14B, faster, and lighter.
- A 14B at naïve 3-bit ties on quality but is slower and heavier, so the extra parameters don’t buy you anything here.
One caveat: comparing a reasoning model (Qwen3) with instruct models is not fully apples-to-apples, since reasoning models trade speed and tokens for accuracy.
The practical takeaway for local deployment: choose by quantization quality, not parameter count. On constrained hardware, a well-quantized smaller model can answer as well as a poorly-quantized larger one while running faster and lighter.
The tool that ran these measurements is open source: ondevice-bench.
I’m Prasad Khake. I work on running LLMs well on on-device hardware. More measurements like this in On Device.