2 posts tagged “qwen3”
I turned on MLX's memory-saving flag and ran out of memory
On a 16 GB Mac, MLX's --kv-bits flag — whose entire job is to shrink the KV cache so longer contexts fit — raised peak memory at every context length I tested, and OOM'd at 32K where plain fp16 fit at 9.4 GB. It's also no faster (8-bit decoding ran ~4× slower in my tests) and costs no quality you'd want to keep. Here's the measurement, the code-level cause, and why the flag backfires on this path.
One flag makes Qwen3-4B beat Llama-3.1-8B on a 16 GB Mac — at half the RAM
On an M3 MacBook Air, Qwen3-4B with the thinking trace turned off scores 20/21 on a verifiable suite — beating Llama-3.1-8B's 18/21 at half the memory and nearly double the speed. With thinking on, the same model drops to 7/21. The flag is enable_thinking=False, and here's exactly what it changes and why it matters.