alt Hacker NewsImpressive debugging skills, and thank you for the benchmarks. Now I'm wondering if mlx-engine / mlx-lm have these bugs too.
One minor thing: as you are concerned with honest numbers, the graphs should be logarithmic on the y-axis too (like they are on the x-axis). Otherwise it's hard to see whether the curve is sublinear or linear.
Thanks, good call. I've switched both throughput charts to a log y-axis (they were already log on x), so the sublinear taper is actually readable now instead of getting flattened by the big prefill numbers up top.
On mlx-engine / mlx-lm: I'd wager it's not resolved upstream. The core bug here is a re-prefill on hybrid recurrent models, and it's not isolated to my setup. oMLX hit it, and llama.cpp has the same issue open right now (https://github.com/ggml-org/llama.cpp/issues/22746). When two independent engines trip on the same thing, it usually points at a shared architectural gap rather than a one-off, so I'd assume mlx-lm is worth checking too.