alt Hacker NewsWe have a lossless compression codec (working on open sourcing it over the next couple of weeks) that reduces it down to its minimum entropy -- it cannot be compressed further. On all tested large models, it's a ratio of 1.34-1.23 -- and smaller models up to 3.76x. It also increases the effective bandwidth by the same rate.
Replies
Alpha3031 • today at 10:35 AM
That's very interesting. Does that mean you can reduce say, a 30B class Q8 from ~30 GB down to 10 GB or less?
➕ show 1 reply
Exploring compression algorithms for weights is a good idea, and I hope you have a successful product. However, if you can prove this statement:
> reduces it down to its minimum entropy -- it cannot be compressed further.
I think you could make a lot more money elsewhere :-)
https://en.wikipedia.org/wiki/Kolmogorov_complexity#Formal_p...