Notes on making LLMs run well on-device — MLX, Apple Silicon, tokenizers, and the seams where models meet real hardware.https://prasadkhake.com/en-ushttps://prasadkhake.com/blog/kv-cache-tax-m3-vs-l4/https://prasadkhake.com/blog/kv-cache-tax-m3-vs-l4/I ran the same long-context test on a 16 GB fanless M3 and a ₹23 rented NVIDIA L4. The laptop fits a 32k context on an 8B model and keeps every planted fact — but prefill balloons to seven minutes and its decode speed can't even be measured, because the fanless chip throttles. A measured, cross-hardware look at the KV-cache tax.Wed, 08 Jul 2026 00:00:00 GMThttps://prasadkhake.com/blog/lora-tool-calling-m3/https://prasadkhake.com/blog/lora-tool-calling-m3/The applied chapter of a from-scratch project: after building tokenization, attention, gradient descent, a tiny GPT, and LoRA by hand, I ran a real QLoRA fine-tune — teaching Llama-3.2-1B to call tools on a MacBook, then measuring honestly what changed and what the adapter costs at inference. A 2.8M-parameter adapter (0.23% of the model) clearly helped on a small test; the debugging taught me the most.Tue, 07 Jul 2026 00:00:00 GMThttps://prasadkhake.com/blog/metal-kernels-from-scratch/https://prasadkhake.com/blog/metal-kernels-from-scratch/I wrote five GPU kernels from scratch on a 16 GB M3 to learn how LLM inference works at the metal. The most useful thing wasn't a kernel — it's a decision rule: never hand-write elementwise ops (the compiler already fuses them), reach for a kernel the moment a reduction appears, and remember the famous trick is rarely the hard part.Tue, 30 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/atomic-turboquant-m3/https://prasadkhake.com/blog/atomic-turboquant-m3/Atomic Chat advertises TurboQuant as 8x faster inference and 6x less memory. I tested the local MLX TurboQuant KV path on a 16 GB M3. It saved about 3-5% total peak memory and did not speed up generation — a useful reminder that KV-cache microbenchmarks do not automatically become whole-chat product claims.Thu, 18 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/self-spec-mlx-m3/https://prasadkhake.com/blog/self-spec-mlx-m3/Self-speculative decoding lets a model draft its own tokens by skipping layers — speculative decoding's speedup with no extra memory. I built it for MLX and swept 24 configs on an M3. Every one was slower than baseline, even though all were lossless. Here's why, and the paper that fixes it.Thu, 18 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/three-ways-fewer-weight-reads-mac/https://prasadkhake.com/blog/three-ways-fewer-weight-reads-mac/Single-stream decoding on Apple Silicon is bottlenecked by reading the model's weights out of memory, not by the math. Three techniques attack that directly — speculative decoding, diffusion generation, and self-speculative layer-skipping. I measured all three on a 16 GB M3. Each is right in theory and backfires in its own way: the bottleneck just moves one step further from the arithmetic.Sun, 14 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/diffusion-llm-16gb-mac/https://prasadkhake.com/blog/diffusion-llm-16gb-mac/LLaDA2.0-mini, a diffusion language model, runs on a 16 GB M3 and ties Qwen3-4B for the best answer-quality score I've measured (20/21). But it's slower than the fastest autoregressive model and uses 4× the memory of the lightest — and the exact reason I expected it to be fast on bandwidth-bound hardware turned out to be why it isn't. A measured look at where the bottleneck actually moved.Sat, 13 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/apple-on-device-model-benchmark/https://prasadkhake.com/blog/apple-on-device-model-benchmark/Apple shipped an official Python SDK for its on-device Foundation Model at WWDC 2026. I put the ~3B model through the same 21-task quality suite I use for MLX models: it ties a 4-bit Llama-3.1-8B (18/21), one question behind Qwen3-4B. Quality is the only fair axis to compare — and that limitation is itself the interesting part.Tue, 09 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/kv-bits-memory-flag-backfires/https://prasadkhake.com/blog/kv-bits-memory-flag-backfires/On a 16 GB Mac, MLX's --kv-bits flag — whose entire job is to shrink the KV cache so longer contexts fit — raised peak memory at every context length I tested, and OOM'd at 32K where plain fp16 fit at 9.4 GB. It's also no faster (8-bit decoding ran ~4× slower in my tests) and costs no quality you'd want to keep. Here's the measurement, the code-level cause, and why the flag backfires on this path.Tue, 09 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/speculative-decoding-16gb-mac/https://prasadkhake.com/blog/speculative-decoding-16gb-mac/A 1B draft model speeds up Llama-3.1-8B by 20% on an M3 — at num_draft_tokens=2. Push that dial to 4 and decoding gets 25% SLOWER than using no draft at all. Here's the measured curve, and why low draft counts win when decode is bound by memory bandwidth.Tue, 09 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/gemma-4-qat-on-m3/https://prasadkhake.com/blog/gemma-4-qat-on-m3/Google's Gemma-4 E4B posts the same math and factual scores as the full 12B on an M3 MacBook Air — in 6.6 GB instead of 11.4, at 8.2 tok/s instead of 2.7 — so on a 16 GB Mac the E4B is the one to run. This is a size win, not a QAT one: the 12B's own QAT build doesn't shrink or speed it up. Honest numbers, measured under a real 2048-token load.Mon, 08 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/benchmark-bugs-that-inflated-my-scores/https://prasadkhake.com/blog/benchmark-bugs-that-inflated-my-scores/Re-running my own LLM benchmark, I found a bug that had inflated the quality scores in posts I'd already published. Then a second bug. Then a third. Here's how a wrong number looks exactly like a right one — and why you spot-check the failures, not the passes.Sun, 07 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/qwen3-4b-thinking-flag-16gb-mac/https://prasadkhake.com/blog/qwen3-4b-thinking-flag-16gb-mac/On an M3 MacBook Air, Qwen3-4B with the thinking trace turned off scores 20/21 on a verifiable suite — beating Llama-3.1-8B's 18/21 at half the memory and nearly double the speed. With thinking on, the same model drops to 7/21. The flag is enable_thinking=False, and here's exactly what it changes and why it matters.Sat, 06 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/gemma-4-12b-m3-benchmark/https://prasadkhake.com/blog/gemma-4-12b-m3-benchmark/Google's Gemma 4 12B uses 11.4 GB of RAM and runs at 2.7 tok/s on an M3 MacBook Air — 2.4× the memory of Llama-3.1-8B at well under half the speed. Its math and factual answers are flawless; its coding can't be cleanly scored. Here's the honest picture, the multimodal tax, and the benchmark bug I found correcting this post.Thu, 04 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/streamingllm-attention-sinks-16gb-mac/https://prasadkhake.com/blog/streamingllm-attention-sinks-16gb-mac/StreamingLLM (2023) found that keeping four specific tokens in the KV cache prevents catastrophic perplexity collapse at long contexts. mlx-lm implements this as RotatingKVCache(keep=4). Here's what that means, why it works, and what our measurements on M3 actually show.Wed, 03 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/gemma-3-12b-vs-qwen3-14b-16gb-mac/https://prasadkhake.com/blog/gemma-3-12b-vs-qwen3-14b-16gb-mac/Benchmarking Gemma-3-12B, Qwen3-14B, and Llama-3.1-8B on a 16 GB MacBook Air (M3) with MLX. A quantization-aware 3-bit 12B ties a naïve 3-bit 14B on overall answer quality while running faster and in less memory. On a memory-bound Mac, a well-quantized smaller model can match a bigger naïvely-quantized one — so parameter count alone is the wrong thing to shop on.Tue, 02 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/16gb-mac-llm/https://prasadkhake.com/blog/16gb-mac-llm/Honest local-LLM benchmarks on a base M3, 16 GB — tokens/sec, peak RAM, and exactly where it hits the wall. The numbers nobody publishes because they run on H100s.Mon, 01 Jun 2026 00:00:00 GMThttps://prasadkhake.com/blog/mlx-tekken-detokenizer/https://prasadkhake.com/blog/mlx-tekken-detokenizer/Debugging why tekken-v13 models emit Ġ instead of spaces through mlx-lm's server, and the one-line root cause in MLX's detokenizer routing.Sat, 30 May 2026 00:00:00 GMT