logoalt Hacker News

Philpaxtoday at 5:17 PM1 replyview on HN

My measurement was with MoE offloading, but there's only so much you can keep on-GPU with a 200GB quant and 48GB of VRAM. It's hard to overcome the CPU/RAM bottleneck.

For what it's worth, all of my hardware was used; I think, all-in, I'm probably at around 3k-4k USD? Not cheap, but also not the worst for something relatively versatile.


Replies

jchwtoday at 6:17 PM

Ah, I see - so MoE offloading is no savior. A shame but no surprise either.