alt Hacker NewsThis is 128B dense though. the K/V cache on long context is going to be massive
Replies
syntaxing • yesterday at 7:22 PM
With turbo quant, you would reduce it by over 6X.
This is 128B dense though. the K/V cache on long context is going to be massive
With turbo quant, you would reduce it by over 6X.
Don’t think kv size correlates to dense/moe