logoalt Hacker News

smallerize • yesterday at 9:54 PM • 1 reply • view on HN

It is already quantized. You need to convert to GGUF format though. Did your assistant explain this wrong?


Replies

drdaeman • yesterday at 9:58 PM

My bad, I haven’t really looked yet (and haven’t asked any assistants), thought it’s BF16 safetensors as usual, and for those I used to quantize to ggufs with just a couple commands.