This sort of thing is a lot of fun.

I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation.

It's been interesting learning about the balance of factors for performant metal kernels on unified memory.

Should have a repo up on github in the next few weeks.