logoalt Hacker News

onlyrealcuzzotoday at 4:36 PM0 repliesview on HN

Standard models are designed to quantize down to 4-bits relatively well.

Anything below that, and especially 1.58b - is typically complete garbage, and you're much better off running a model 100x smaller at regular precision (compared to one 7x smaller quantized into complete garbage).

If the model was designed specifically to quantize down to 1.58b, then it's different.

AFAIK, there's no large models designed for this yet.