alt Hacker NewsThe most counter-intuitive bug was that a unique message ID in the system prompt broke the entire KV cache. Since the cache requires byte-exact matches, that changing ID forced a full re-compute on every turn, turning warm contexts into cold fills.
I've open-sourced the fork (qMLX) and a benchmark script (bench_qmlx.py) that separates prefill/decode metrics. I chose to fork rather than submit a PR because these hybrid attention changes are specific to the Qwen flavor of models and would likely be unpalatable to upstream maintainers who prioritize a general-purpose stack. I expect this fork to continue diverging from the base as we optimize specifically for this architecture. Happy to answer questions about the caching strategy or eviction logic.
Replies
> The most counter-intuitive bug was that a unique message ID in the system prompt broke the entire KV cache.
How is that counter-intuitive? You changed the key, so it didn't return a value. Thats a KV cache working as intended.
> a unique message ID in the system prompt broke the entire KV cache. Since the cache requires byte-exact matches, that changing ID forced a full re-compute on every turn, turning warm contexts into cold fills.
This is (part of) the same problem that initially lead Anthropic to ban non-Claude Code clients from using the subsidized subscription: A full to-the-second datetime stamp in the system prompts of OpenCode, and I believe Pi as well, invalidated the caches, making this a very expensive use of their compute very quickly.
They even had Anthropic employees submit PRs (or maybe just open issues, I’d have to check) to these other clients/harnesses because the cache misses were hitting them so hard.