At 32,000 tokens, the costliest thing my MacBook did was wait seven minutes to speak
At 32,000 tokens of context, the loudest cost on my 16 GB MacBook was not the answer. It was the seven-minute wait before the model said a single word.
I wanted to know exactly what a laptop can and cannot do with long context, so I ran the same test on two machines: my fanless M3 MacBook, and an NVIDIA L4 I rented from Jarvislabs, an Indian GPU cloud, for ₹23 (about thirty cents an hour, for half an hour). Same 8B model. Same growing document. The numbers below are all measured, warmed, and repeated three times. None of them are estimates.
The tax is in prefill. At 32k the M3 waits about seven minutes for the first token; the L4 waits fifteen seconds (left, log scale). Decode is where the fanless M3 stops being measurable at all: it throttles under sustained load, so its line is greyed as observed-not-clean, while the actively cooled L4 draws the curve properly (right).
What the laptop is good at
Two things held up, and they answer the first question anyone actually has, which is “can I even do this on my own machine.”
It fits. Peak memory rose from 5.3 GB at 2,000 tokens to 9.3 GB at 32,000. Most of that extra four gigabytes is the KV cache, the model’s running memory of the conversation, which grows with every token you add. It never touched the 16 GB ceiling at 32k, though 64k probably would.
The facts survive. I planted five made-up access codes near the front of the document and asked for all five at the end. The model returned 5 out of 5 at every length, on both machines. A full-precision KV cache holds early context perfectly well. (This is an early-needle test, not facts scattered through the whole document. That harder test is the next one.)
So: an 8B model fits on a 16 GB laptop at 32k, and it remembers what you told it. Good news.
The bill comes due at prefill
The cost is not in generating the answer. It is in prefill, the one-time read through the whole prompt before the first token appears. On the M3 that grew from 20 seconds at 2k to 430 seconds at 32k. Seven minutes. That is roughly 22 times slower for 16 times the context, so it worsens faster than the context grows. You pay it every time you feed in a long prompt without a reusable prefix cache.
The L4 pays the same-shaped bill about thirty times cheaper: 0.8 seconds to 15 seconds across the same range. That single gap is why a long-context agent feels unusable on a laptop and perfectly fine in the cloud. It is the wait before anything happens at all.
The number I could not measure, which became the point
I set out to draw a clean line of decode speed against context length on the M3. I could not. And finding out why was the most useful half hour of the whole exercise.
The decode numbers came out backwards: 8k slower than 16k, which is impossible, because more context means more memory to read per token, which is always slower. It was not a bug. The exact same client code drew a flawless line on the L4. It was heat. Forty minutes of continuous inference on a fanless chip throttles the entire machine, and that drift was larger than the effect I was trying to measure.
So I tested it head-on. Run a small 2k task cold. Heat the chip with a 32k run. Immediately rerun the identical 2k task. Three times, then let it cool and try once more.
| cycle | cold 2k | hot 2k, right after a 32k run | drop |
|---|---|---|---|
| 1 | 18.4 tok/s | 15.6 | 15% |
| 2 | 16.5 | 9.5 | 42% |
| 3 | 17.4 | 10.8 | 38% |
| after an 8-minute cooldown | 18.9 | recovered |
The slowdown is repeatable, it happens every cycle, and it is reversible, because after cooling the same task climbs back to 18.9 tok/s, past where the first cycle even started. The system’s own power readout showed the CPU clocks sliding from about 3.2 GHz down toward 1.1 GHz under load. That is thermal throttling, plainly.
Which leads to the honest conclusion. On a fanless M3 under sustained load, decode speed against context length is not a line you can draw truthfully, because the machine slows itself down before it finishes. The L4, with a fan, drew it without complaint: 50, 44, 38, 29 tokens per second.
That contrast is the real finding. A fanless laptop will tell you whether a model fits, how painful prefill gets, and whether recall survives. It will not tell you sustained throughput, because it throttles first. That is not a defect. It is a boundary worth knowing before you trust a benchmark run on one.
Two runtimes, two definitions of peak memory
A warning for anyone comparing hardware. The two runtimes account for memory in opposite ways. MLX, on the Mac, grows the KV cache as it goes; I watched it reach 9.3 GB. vLLM, on the L4, reserves it up front: it claimed 13.4 GiB for the cache before serving a single request, enough for about 110,000 tokens. So “peak GB” is not the same quantity on both machines. One is what was used. The other is what was set aside. The honest shared number is memory per token, roughly 128 KiB per token for this model, which matches vLLM’s own reservation exactly (13.37 GiB across 109,504 tokens).
Why any of this matters
The bottleneck under all of it is moving bytes, not doing arithmetic. Every single token, the chip must re-read the model’s weights and the growing KV cache out of memory. That is why prefill dominates, why decode is limited by memory bandwidth rather than compute, and why a cheap GPU with a fan beats a laptop on the one axis the laptop cannot hold still.
It is the same story being told at every scale right now, from wafer-scale accelerators to the fresh idea of streaming model weights straight off flash. The KV cache is the awkward character in that story. Unlike the weights, it grows, it is rewritten on every token, and it is the thing that actually runs a laptop out of room.
This is the first slice of a larger cross-hardware map I am building. Next: KV eviction and compression, and precisely how much recall you surrender to buy back the prefill and the memory.
The code and the raw data from both machines are here: github.com/robertlangdonn/kv-cache-tax.