alt Hacker NewsWhy is anyone even using video cards these days? You may as well be burning cash.
This is the perfect candidate for just splattering it on your nvme and then reading it off there and into memory. All of these run perfectly fine on simple m4 silicone:
https://github.com/drumih/turbo-fieldfare
Those are all for MoE models. And I prefer measuring my tokens in t/s instead of s/t