logoalt Hacker News

syntaxingtoday at 12:53 AM0 repliesview on HN

Q8 or Q6_UD with no KV cache quantization. I swear it matters even more with small activated parameters MOE model despite the minimal KL divergence drop