logoalt Hacker News

Wheentoday at 7:32 PM0 repliesview on HN

Sorta, but you're off by one layer. You can store the 18B in VRAM and stream the rest from RAM. There's still a performance hit relative to storing it all in VRAM, but it's tolerable.

Generally, for local consumer use, these large MOE models are best for unified RAM systems like DGX Spark or Mac Studio.