MoE layers offload to CPU inference is the easiest way, though a bit of a drag on performance
Yeah, I'd just be pretty surprised if they were getting 100 tokens/sec that way.
EDIT: Either they edited that to say "quad 3090s", or I just missed it the first time.
alt Hacker News
Yeah, I'd just be pretty surprised if they were getting 100 tokens/sec that way.
EDIT: Either they edited that to say "quad 3090s", or I just missed it the first time.