alt Hacker NewsAnything good to run on Mac M5 Max with 48GB? is this even worth trying? so far I found the responses so slow compared to the paid subscriptions...
Anything good to run on Mac M5 Max with 48GB? is this even worth trying? so far I found the responses so slow compared to the paid subscriptions...
I get up to 90 tokens / second with Jundot/Qwen3.6-35B-A3B-oQ6-mtp, on a M3 Max with 64GB. MoE so it is not a dense model, but that means it runs faster (also, mtp helps). It is a 30GB model, but you should be able to load it, otherwise try the 4-bit quant instead of the 6-bit quant, don't bother quanting your KV Cache (don't enable turboquant in oMLX), since that will slow you down.
I'm not sure what that means on a M5 max, definitely faster, I don't know if it really plays into the strengths of the new chip design though.