alt Hacker NewsEven with inflated RAM prices, you can buy a Strix Halo Mini PC with 128GB unified memory right now for less than 2k. It will run gpt-oss-120b (59 GB) at an acceptable 45+ tokens per second: https://github.com/lhl/strix-halo-testing?tab=readme-ov-file...
I also believe that it should eventually be possible to train a model with somewhat persistent mixture of experts, so you only have to load different experts every few tokens. This will enable streaming experts from NVMe SSDs, so you can run state of the art models at interactive speeds with very little VRAM as long as they fit on your disk.
I agree the parent is a bit too pessimistic, especially because we care about logical skills and context size more than remembering random factoids.
But on a tangent, why do you believe in mixture of experts?
Every thing I know about them makes me believe they're a dead-end architecturally.