logoalt Hacker News

zozbot234yesterday at 4:49 PM0 repliesview on HN

That "traditional" setup is the recommended setup for running large MoE models, leaving shared routing layers on the GPU to the extent feasible. You can even go larger-than-system-RAM via mmap, though at a non-trivial cost in throughput.