Apple's on-device model ties a 4-bit Llama-3.1-8B — and won't name the M1
Apple’s on-device Foundation Model — the ~3B model that just exists in macOS — ties a 4-bit Llama-3.1-8B on answer quality, and sits one question behind a Qwen3-4B you’d download yourself. I measured it through the official Python SDK Apple shipped at WWDC 2026, on the same 21-task verifiable suite I run against MLX models. For a model you never download, quantize, or manage — it’s just part of the operating system — that’s a genuinely strong showing.
Which model is this? (clarification, 2026-06-09): This is the on-device model shipping in macOS 26 — the generation before the AFM 3 family (AFM 3 Core, and the 20B-sparse AFM 3 Core Advanced) that Apple announced at WWDC 2026. AFM 3 ships with macOS 27, and I haven’t put that beta on my only machine — so these are numbers for the model you can run on a released Mac today, not the new one. The new SDK runs on macOS 26 and serves the current-gen model; it doesn’t expose a version string, so the distinction is by OS. Benchmarking AFM 3 (especially the sparse Core Advanced) is the sequel once macOS 27 ships.
Quality on one uniform 21-task check (8 coding, 8 math, 5 factual). Apple’s on-device 3B (18/21) ties the 4-bit Llama-3.1-8B and trails the Qwen3-4B champion by a single question — while being the only model here you don’t have to install.
Until this week, using Apple’s on-device model from Python meant a community wrapper. At WWDC 2026 Apple shipped python-apple-fm-sdk — official bindings to the on-device Foundation Model, the one that powers features like Writing Tools. On macOS 26 that’s the current-generation model (not AFM 3 — see the note above); it runs in-process, no API key, no network. So I pointed my benchmark at it.
The quality numbers, same suite, same machine (M3):
| Model | Coding | Math | Factual | Total |
|---|---|---|---|---|
| Qwen3-4B (thinking off) | 8/8 | 8/8 | 4/5 | 20/21 |
| Apple Foundation (3B) | 7/8 | 7/8 | 4/5 | 18/21 |
| Llama-3.1-8B (4-bit) | 8/8 | 5/8 | 5/5 | 18/21 |
Apple’s 3B and the 8B Llama land on the same total by different routes: Apple is stronger at math (7/8 vs 5/8), Llama is cleaner on coding and factual recall. A tie on the bottom line, from a model less than half the size of one and built into the OS.
What it actually got wrong
Three real misses, and I read every one to be sure they weren’t my benchmark lying again (more on that below):
- A string reversal it broke. Asked to reverse
'OpenSource', it wroteprint(OpenSource[::-1])— forgot the quotes, referenced an undefined variable, crashed. Correct idea, broken code. - A percentage it flubbed. “What is 17.5% of 240?” → it answered 40.5. The answer is 42.
- Its own chip. Asked for “Apple’s first in-house Mac chip, released 2020,” it answered “Apple Silicon” — the brand, not the M1. The one model in this lineup made by the company that built the M1 is the one that wouldn’t name it.
Why I only compare quality
This is the honest core of the post: quality is the only axis on which this comparison is fair. With an MLX model I run, I measure exactly what it costs — peak RAM with mx.get_peak_memory(), decode tokens/sec from the tokenizer. Apple’s model gives me neither. It runs in a system process on the Neural Engine that my Python process doesn’t own, so I can’t see its memory the way I see an MLX model’s, and the SDK doesn’t expose a token count, so I can’t compute a real tokens/sec. I could print a number, but it wouldn’t mean the same thing as the MLX figures, and putting them in the same column would be the kind of false-equivalence this benchmark exists to avoid. So: quality head-to-head, measured identically; cost and speed, deliberately left out.
That opacity is itself the finding. The trade you make for “it’s already there and it’s private” is that it’s a black box — you can’t profile what it costs you the way you can with a model you run yourself.
The benchmark bug I found doing this
Apple’s model first scored 17/21, with fizzbuzz marked failed. The code was correct — but my checker required the printed output’s last line to end with "FizzBuzz", and a printed Python list ends with ']. It was scoring output format, not correctness. The fix bumped Apple to 18/21 — and, because every model here prints the list the same way, it bumped Qwen3-4B to 20/21 and Llama to 18/21 too. The ranking didn’t change; the floor moved up by one. It’s the fourth bug I’ve found in my own benchmark, and the lesson is the one that keeps repaying: spot-check the failures, not the passes — a surprising failure is as likely to be your harness as the model.
The upshot
The on-device model you didn’t know you had is competitive with an 8B you’d manage yourself. It won’t replace a Qwen3-4B you run deliberately — that’s still the quality-per-gigabyte champion — but for an app that wants private, zero-setup language smarts, “as good as a local Llama-3.1-8B, already installed” is a strong default. And this is the current on-device model — the new AFM 3 Core, and especially the 20B-sparse Core Advanced, are what I want to put through this suite once macOS 27 is something I’ll run on a primary machine.
(Caveats, stated plainly: this is a ~3B task-tuned model run at the SDK’s default settings, one pass. The harness records any guardrail refusal as a task error rather than a silent fail — though in this run all 21 tasks returned real answers, no refusals. The quality numbers are the robust claim; treat them as a snapshot, not a leaderboard.)