alt Hacker NewsUse modern samplers and you don’t need to limit yourself to 8bit at half the context window. I could push it down to 1.58 bits and get decently good output easily by simply not using the garbage default top_p and top_k that vendors continue to wrongly recommend.
Where do you find optimal samplers and sampler settings for these models? Very interested in this as I, too, use Q8.