I thought Q4_K_M is the standard. Why did you choose the 6-bit variant? Does it generate better input?
There is no standard.
The higher quantization - the better results, but more memory is needed. Q8 is the best.
alt Hacker News
There is no standard.
The higher quantization - the better results, but more memory is needed. Q8 is the best.