alt Hacker NewsYes but most people are still running MoE models with all experts loaded in RAM! This experiment shows quite clearly that some experts are only rarely needed, so you do benefit from not caching every single expert-layer in RAM at all times.
Replies
jnovek • yesterday at 5:54 PM
I’m so confused in these comments right now — I thought you had to load an entire MoE model and sparseness just made it so you can traverse the model more quickly.
MillionOClock • yesterday at 7:25 PM
I hope some company trains their models so that expert switches are less often necessary just for these use cases.
➕ show 1 reply
That's not what this test shows. It's just loading the parts of the model that are used in an on-demand fashion from flash.
The iPhone 17 Pro only has 12GB of RAM. This is a -17B MoE model. Even quantized, you can only realistically fit one expert in RAM at a time. Maybe 2 with extreme quantization. It's just swapping them out constantly.
If some of the experts were unused then you could distill them away. This has been tried! You can find reduced MoE models that strip away some of the experts, though it's ony a small number. Their output is not good. You really need all of the experts to get the model's quality.