alt Hacker NewsAt which quantization? That's the only thing I want to know.
EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.
Replies
I’m confused too and the slop readme doesn’t help at all.
> Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB
And later
> Standard IQ2_XXS (76.87 GiB, Unpatched)
Except unsloth’s IQ2_XXS GGUF is 91GB, even their smallest IQ1_S is 82.5GB. So I wonder which model/quant the number even refers to.
Also the mentions of “compression” and “MoE pruning” also make it sound lobotomized even more than unsloth’s IQ1 quants.
This 100%
Reddit is flooded with these breathless headlines “xyz tokens/s on my light switch”. It’s always quantizations below 4 bits per weight. Mostly useless.
Everything lower than 4/5 is really bad