Learning fine-tuning by building a tool-calling LoRA on an M3
This is the applied chapter of a from-scratch project. Over the last stretch I’ve been learning
how LLMs actually work by building the pieces myself — tokenization, attention, gradient descent,
a tiny GPT, and LoRA from scratch — and this post is the test of whether that understanding
survives contact with a real fine-tune. So I taught Llama-3.2-1B-Instruct to call tools on my
MacBook with LoRA (QLoRA-style, over a 4-bit base), then measured what changed and what the
adapter costs to run.
The short version: a 2.8M-parameter adapter — 0.23% of the model — clearly helped on a small held-out test, in about 18 minutes on an M3 Air. It’s a toy on purpose; the goal was to understand each piece end to end. And the debugging along the way — a bad learning rate, a contaminated test split, a benchmark harness that lied — taught me more than the result did.
What I was trying to teach
I gave Llama-3.2-1B-Instruct (4-bit) three make-believe tools — get_weather(city),
set_reminder(time, task), web_search(query) — and wanted it to do two things: emit a clean
tool call when one’s needed, and not call a tool for plain chit-chat (“who painted the Mona
Lisa?”). That second part — knowing when to stay quiet — turned out to be the hard bit.
The training data is 102 tiny examples in mlx-lm’s tools format, plus a held-out test set. I
kept the test set’s cities, queries, and chit-chat separate from training, so the score would be
a better test than exact-string memorization (still only narrow generalization within the same
toy distribution). More on that promise below — I got it wrong the first time.
A 0.23%-of-the-model adapter improved tool-calling on this toy test — the base model called
get_weather("Tokyo") for almost everything it was unsure about, scoring 0/5 on knowing when
not to call a tool. (Kept separate, that adapter costs ~15–20% decode speed; fused, it costs a
680 MB weight copy instead of an 11 MB adapter — more on that below.)
The before / after
On a 15-example held-out test set:
| Base model | After LoRA | |
|---|---|---|
| Tool calls correct | 6/10 | 9/10 |
| Knew not to call a tool | 0/5 | 4/5 |
| Overall | 6/15 (40%) | 13/15 (87%) |
The base model’s failure was funny and consistent: when unsure, it just called
get_weather("Tokyo") for everything — including “who wrote Hamlet?” and “that’s all for now.”
It had no instinct for when a tool wasn’t wanted (0/5). After fine-tuning, it mostly learned to
stay quiet on chit-chat and to pick the right tool.
It’s 15 examples, so I read this as “the fine-tune clearly helped on this toy,” not “87% accuracy” as if that were a benchmark. Each example is worth ~7 points; I’m not going to pretend that’s precise. The adapter that did this was 2.8 million parameters — 0.23% of the model — and training took about 18 minutes on an M3 Air, inside ~5 GB of RAM.
The failures, which taught me more than the wins
My first training run got worse, not better. Instead of falling, the validation loss climbed. I’d set the learning rate too high, so each update overshot — the same thing you can watch happen in a ten-line gradient-descent toy. Dropping it from 2e-4 to 1e-4 fixed it: the loss fell to ~0.01. Here’s the same data trained twice; the only difference is the step size:
Same model, same data — the only change is the learning rate. At 1e-4 (blue) the loss drops to ~0.01 and stays there. At 2e-4 (rust) each step overshoots the target, so the loss climbs past 6 and never recovers. “Loss going up” isn’t always a deep problem; sometimes you just stepped too far.
Two of the after-fine-tune mistakes are the interesting kind. “catch the train for 6:30am”
came back with the time as 6am — it dropped the :30. And “who painted the Mona Lisa?”
triggered a web_search instead of a plain answer — which, honestly, is a reasonable thing to do;
I’d just labeled factual questions as no-tool. A clean-looking 87% still hides judgment calls like
that.
The bug I didn’t catch — but a second model did
Before publishing I ran the whole thing past a second model (OpenAI’s Codex CLI) and asked it to find holes. It found a real one immediately: my “no-tool” test was contaminated. Three of the four chit-chat test examples were exact copies of training examples. So my headline “it learned when not to call a tool” was mostly measuring memorization, not learning. Embarrassing, and true.
I fixed it — split the chit-chat into disjoint train and test pools, so the test phrases are ones the model never saw — and re-ran everything. The numbers above are the honest, post-fix version. The no-tool score (0/5 → 4/5) now really is on unseen chit-chat.
It also flagged that my inference-cost script was lying (below), that holding out entity values but reusing the same three tools only shows narrow “fill in the slot” generalization, and that my eval is a simple string matcher, not a robust parser. All fair. Running a second model over your own work before you publish is cheap, and it catches the stuff you’re too close to see.
What the adapter costs at inference
This is the part most fine-tuning tutorials skip, and the bit I actually cared about. A LoRA adapter can run two ways, and they trade off:
| Setup | Decode speed | Peak RAM | On disk |
|---|---|---|---|
| Base only | ~102 tok/s | 0.80 GB | — |
| Base + adapter (kept separate) | ~84 tok/s | 0.82 GB | +11 MB |
| Fused (adapter merged in) | ~101 tok/s | 0.80 GB | 680 MB |
Kept separate, the adapter costs roughly 15–20% of decode speed — each adapted layer does two extra small matrix multiplies per token. Fusing it into the weights (precomputing the merged weight) gets that speed back, because the merged model is shaped exactly like the base. But fusing ships a full 680 MB copy; the separate adapter is 11 MB and you can hot-swap it. So: speed and footprint vs. flexibility. (Caveat: this is a quick single-machine measurement, a few runs on a fanless M3 — treat it as “roughly 15–20%,” not a precise benchmark.)
The lying-harness bug, for the record: my first cost script divided a fixed token count by wall time, but the fine-tuned model emits a short tool call and stops early — so the division reported it running 3× faster than the base, which is impossible. Counting the tokens actually generated fixed it. The measurement harness is usually the least-tested code in any benchmark.
One thing worth being precise about: this is QLoRA-style (LoRA over a 4-bit quantized base). The base stays frozen and quantized, and only the small adapter trains. It does not train 4-bit weights directly; the quantized weights are dequantized on the fly for the matmuls, and the little adapter learns on top. That’s what keeps the whole thing inside a laptop’s memory.
What I actually took away
- Fine-tuning a specific behavior onto a capable base is genuinely cheap — a fraction of a percent of the weights, minutes on a laptop.
- “Cheap to train” isn’t “free to run” — in this setup, budget the ~15–20% adapter tax, or fuse and budget the disk.
- The failures (bad learning rate, contaminated split, a lying benchmark harness) taught me more than the success did, and most of them are invisible until you go looking — or ask another model to look.
Code — dataset, training, eval, and the cost harness — is here: github.com/robertlangdonn/lora-toolcalls.