alt Hacker NewsI know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps excessively so. I wonder, is there a way to make the RAM usage tunable? I have a Macbook with 32 GB of RAM, and it'd be great if I could run the same model but take advantage of the additional RAM to make it run faster.
Replies
ianmurrays • today at 4:50 AM
I guess you have to know which experts to keep “hot” in ram, which you can’t know beforehand, so there wouldn’t be much gain.
➕ show 1 reply
That's about the turning point for just using typical quants for me. Larger still and you can just do the full model. Smaller to this degree and you need all sorts of extra tricks to get anything.