Unsloth use a property dataset they don't release, however you can indeed create quantisation locally on your machine and it's pretty easy, llama.cpp comes with everything you need.

Just quantizing takes seconds-to-minutes, llama.cpp provides a nice tool[0]. Improving quality is then a matter of picking specific tensors to maintain at higher accuracy, checking on representative data, and repeating.

[0]: https://github.com/ggml-org/llama.cpp/blob/master/tools/quan...

Quantization is typically very cheap and fast. It can even be done on hardware that does not fit the model, by processing the weights layer by layer.

I use this project: https://github.com/vllm-project/llm-compressor