2 posts tagged “llama”
I turned on MLX's memory-saving flag and ran out of memory
On a 16 GB Mac, MLX's --kv-bits flag — whose entire job is to shrink the KV cache so longer contexts fit — raised peak memory at every context length I tested, and OOM'd at 32K where plain fp16 fit at 9.4 GB. It's also no faster (8-bit decoding ran ~4× slower in my tests) and costs no quality you'd want to keep. Here's the measurement, the code-level cause, and why the flag backfires on this path.
Speculative decoding on a 16 GB Mac: a 20% win that becomes a 25% loss
A 1B draft model speeds up Llama-3.1-8B by 20% on an M3 — at num_draft_tokens=2. Push that dial to 4 and decoding gets 25% SLOWER than using no draft at all. Here's the measured curve, and why low draft counts win when decode is bound by memory bandwidth.