3 posts tagged “inference”
At 32,000 tokens, the costliest thing my MacBook did was wait seven minutes to speak
I ran the same long-context test on a 16 GB fanless M3 and a ₹23 rented NVIDIA L4. The laptop fits a 32k context on an 8B model and keeps every planted fact — but prefill balloons to seven minutes and its decode speed can't even be measured, because the fanless chip throttles. A measured, cross-hardware look at the KV-cache tax.
I built self-speculative decoding for MLX. On an M3, naive layer-skip never beats baseline — 24 configs, 24 losses
Self-speculative decoding lets a model draft its own tokens by skipping layers — speculative decoding's speedup with no extra memory. I built it for MLX and swept 24 configs on an M3. Every one was slower than baseline, even though all were lossless. Here's why, and the paper that fixes it.
Speculative decoding on a 16 GB Mac: a 20% win that becomes a 25% loss
A 1B draft model speeds up Llama-3.1-8B by 20% on an M3 — at num_draft_tokens=2. Push that dial to 4 and decoding gets 25% SLOWER than using no draft at all. Here's the measured curve, and why low draft counts win when decode is bound by memory bandwidth.