alt Hacker NewsSo about 24gb/expert at NVFP4. If you compressed the KV cache, you might be able to get away with 2x 24GB cards and stream the experts.
So about 24gb/expert at NVFP4. If you compressed the KV cache, you might be able to get away with 2x 24GB cards and stream the experts.
If you're happy with generation speed in 5 t/s range, sure -- PCIe 5.0 x16 is only 64GB/s.