alt Hacker NewsI'll pipe in - a series of Mac optimized MOEs which can stream experts just in time would be really amazing. And popular; I'm guessing in the next year we'll be able to run a very able openclaw with a stack like that. You'll get a lot of installs there. If I were a PM at Gemma, I'd release a stack for each Mac mini memory size.
Expert streaming is something that has to be implemented by the inference engine/library, the model architecture itself has very little to do with it. It's a great idea (for local inference; it uses too much power at scale), but making it work really well is actually not that easy.
(I've mentioned this before but AIUI it would require some new feature definitions in GGUF, to allow for coalescing model data about any one expert-layer into a single extent, so that it can be accessed in bulk. That's what seems to make the new Flash-MoE work so well.)