Apple's on-device model ties a 4-bit Llama-3.1-8B — and won't name the M1
Apple’s on-device Foundation Model — the ~3B model that just exists in macOS — ties a 4-bit Llama-3.1-8B on answer quality, and sits one question behind a Qwen3-4B you’d download yourself. I measured it through the official Python SDK Apple shipped at WWDC 2026, on the same 21-task verifiable suite I run against MLX models. For a model you never download, quantize, or manage — it’s just part of the operating system — that’s a genuinely strong showing.
Which model is this? (clarification, 2026-06-09): This is the on-device model shipping in macOS 26 — the generation before the AFM 3 family (AFM 3 Core, and the 20B-sparse AFM 3 Core Advanced) that Apple announced at WWDC 2026. AFM 3 ships with macOS 27, and I haven’t put that beta on my only machine — so these are numbers for the model you can run on a released Mac today, not the new one. The new SDK runs on macOS 26 and serves the current-gen model; it doesn’t expose a version string, so the distinction is by OS. Benchmarking AFM 3 (especially the sparse Core Advanced) is the sequel once macOS 27 ships.
Quality on one uniform 21-task check (8 coding, 8 math, 5 factual). Apple’s on-device 3B (18/21) ties the 4-bit Llama-3.1-8B and trails the Qwen3-4B champion by a single question — while being the only model here you don’t have to install.
Until this week, using Apple’s on-device model from Python meant a community wrapper. At WWDC 2026 Apple shipped python-apple-fm-sdk — official bindings to the on-device Foundation Model, the one that powers features like Writing Tools. On macOS 26 that’s the current-generation model (not AFM 3 — see the note above); it runs in-process, no API key, no network. So I pointed my benchmark at it.
The quality numbers, same suite, same machine (M3):
| Model | Coding | Math | Factual | Total |
|---|---|---|---|---|
| Qwen3-4B (thinking off) | 8/8 | 8/8 | 4/5 | 20/21 |
| Apple Foundation (3B) | 7/8 | 7/8 | 4/5 | 18/21 |
| Llama-3.1-8B (4-bit) | 8/8 | 5/8 | 5/5 | 18/21 |
Apple’s 3B and the 8B Llama land on the same total by different routes: Apple is stronger at math (7/8 vs 5/8), Llama is cleaner on coding and factual recall. A tie on the bottom line, from a model less than half the size of one and built into the OS.
What it actually got wrong
Three real misses, and I read every one to be sure they weren’t my benchmark lying again (more on that below):
- A string reversal it broke. Asked to reverse
'OpenSource', it wroteprint(OpenSource[::-1])— forgot the quotes, referenced an undefined variable, crashed. Correct idea, broken code. - A percentage it flubbed. “What is 17.5% of 240?” → it answered 40.5. The answer is 42.
- Its own chip. Asked for “Apple’s first in-house Mac chip, released 2020,” it answered “Apple Silicon” — the brand, not the M1. The one model in this lineup made by the company that built the M1 is the one that wouldn’t name it.
Why I only compare quality
This is the honest core of the post: quality is the only axis on which this comparison is fair. With an MLX model I run, I measure exactly what it costs — peak RAM with mx.get_peak_memory(), decode tokens/sec from the tokenizer. Apple’s model gives me neither. It runs in a system process on the Neural Engine that my Python process doesn’t own, so I can’t see its memory the way I see an MLX model’s, and the SDK doesn’t expose a token count, so I can’t compute a real tokens/sec. I could print a number, but it wouldn’t mean the same thing as the MLX figures, and putting them in the same column would be the kind of false-equivalence this benchmark exists to avoid. So: quality head-to-head, measured identically; cost and speed, deliberately left out.
That opacity is itself the finding. The trade you make for “it’s already there and it’s private” is that it’s a black box — you can’t profile what it costs you the way you can with a model you run yourself.
The benchmark bug I found doing this
Apple’s model first scored 17/21, with fizzbuzz marked failed. The code was correct — but my checker required the printed output’s last line to end with "FizzBuzz", and a printed Python list ends with ']. It was scoring output format, not correctness. The fix bumped Apple to 18/21 — and, because every model here prints the list the same way, it bumped Qwen3-4B to 20/21 and Llama to 18/21 too. The ranking didn’t change; the floor moved up by one. It’s the fourth bug I’ve found in my own benchmark (the first three are their own post), and the lesson is the one that keeps repaying: spot-check the failures, not the passes — a surprising failure is as likely to be your harness as the model.
The upshot
The on-device model you didn’t know you had is competitive with an 8B you’d manage yourself. It won’t replace a Qwen3-4B you run deliberately — that’s still the quality-per-gigabyte champion — but for an app that wants private, zero-setup language smarts, “as good as a local Llama-3.1-8B, already installed” is a strong default. And this is the current on-device model — the new AFM 3 Core, and especially the 20B-sparse Core Advanced, are what I want to put through this suite once macOS 27 is something I’ll run on a primary machine.
(Caveats, stated plainly: this is a ~3B task-tuned model run at the SDK’s default settings, one pass. The harness records any guardrail refusal as a task error rather than a silent fail — though in this run all 21 tasks returned real answers, no refusals. The quality numbers are the robust claim; treat them as a snapshot, not a leaderboard.)