alt Hacker NewsIsn't Q8 way overkill these days? I see many graphs showing Q4 or Q5 having less than %1 deviation. Nvidia's NVFP4 Qwen quantization should be even better due to its better training methods.
Replies
mdgld • today at 5:05 PM
It depends on model size I think, but yeah, from my understanding at ~30B and below Q6 or even Q4 will get you 95%+ of the way there
When orgs/bencmarks claim 1% deviation, in most cases that means measuring perplexity loss on datasets like wikitext or c4. Even if the loss is calculated via KLD or similar, its not a good proxy for whats actually degradaing at the task level across an entire rollout.
And for MoEs, very small amounts of loss can mean you're flipped to entirely different experts (this is also a problem more broadly with numerical stability issues too).