> Even if the output is like 5-6 tok/s
On a 3T model I’d imagine you’d be closer to 0.05 tks
Presumably it’s MoE and only needs to read a small fraction of the weights per token. Bonus points if you can get decent speculative decoding without becoming ALU-limited.
alt Hacker News
Presumably it’s MoE and only needs to read a small fraction of the weights per token. Bonus points if you can get decent speculative decoding without becoming ALU-limited.