alt Hacker News> running it on a no GPU, but tons of RAM server
Or from SSD using something like Colibri[1]. Not going to be quick, but at least runable.
> running it on a no GPU, but tons of RAM server
Or from SSD using something like Colibri[1]. Not going to be quick, but at least runable.
It's a great concept but I think it would cross the line from 'very slow' to 'so slow it's unusable' at this size. Even if we say you have an NVME SSD that does 7GB/s reads, that's dramatically slower than being able to hold the whole thing in DRAM. Like the difference between 1.3 tok/s in RAM vs 0.1 tok/s with a colibri-like method.
edit: the results I have seen from people trying colibri with fast consumer grade PCI-E 4.0 NVME SSD are 0.1 tok/s on models that are <700B in size, things that are well under 800GB on disk. With something that's 3T in size it'll probably be a lot slower than hat.