logoalt Hacker News

pizza234today at 1:55 PM2 repliesview on HN

> but are they functionally useful or helpful to anyone currently?

Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do.

MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro.

The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).


Replies

ddevnyctoday at 2:25 PM

Does MoE help with multimodality? Can it in general enable reasoning in imagery (technical drawings, diagrams, schematics) rather than text-based?

show 1 reply
bensyversontoday at 2:27 PM

Are people getting decent tokens/second throughput? Some of these demos crawl at 1 tok/s or worse, which limits their utility.

show 2 replies