This mentality is really pervasive and yet I've found when a lower than 4-bit quant of a given model was all I could use, then using that was often better than using a 4-bit quant of a model that I could. YMMV. It just really made me rethink my stance on the whole thing. Most people likely see something like 2-bit and just feel like the want to puke without ever actually learning whether or not it can handle their tasks and how well relative to other models they run. I've been running Qwen3.8-Flash-Next at Q4, but was crazy surprised at the performance of Unsloth's IQ1_M of DeepSeek v4.1 Flash.

Deepseek v4 flash 0731 in a 2-bit quant has been useful for me. Other models don't seem to hold up nearly as well at 2-bit. Most of my other models are 5-6 bit quants because of this.

Well it's bad compared to the original model, but is it bad compared to what else you could run in 64 GB?

This but also the kv quantization shouldn’t be lower than 8 bits.

Next frontier is achieving good output with that kind of quant dimension

There are solutions for that: https://prismml.com/news/bonsai-2-27b

(and before you crap on it, this is not the same as the original Bonsai, this newer one is much better, and upstream support in llama.cpp etc. is being worked on)