logoalt Hacker News

Lio • today at 10:54 AM • 7 replies • view on HN

RAM compression sounds very useful but the thing I really need extra RAM for is local inference.

Since the models are effectively highly compressed data already my assumption is that RAM compression won't help improve that much (I'm assuming unified memory here as per the Framework Desktop). A bit like trying to compress a JPEG in a zip file.

Could anyone comment on that?


Replies

throw-qqqqq • today at 3:55 PM

The max theoretical ratio of (byte-independent) compression can be approximated using the Shannon entropy.

Most quantized GGUFs have a very high Shannon entropy, so are almost perfectly compressed already. A Q4_K_M quantization is often around +7.90 bits/byte and so is virtually incompressible.

Unquantized models (F16) are often around +7bits/byte for comparison.

https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theo...

hobo123 • today at 8:03 PM

Since you need to apply CPU time to compress memory, and also CPU before you can map pages back into process memory (by uncompressing), this is only useful for not frequently used memory, like swap only a bit faster.

I'm not too familiar with LLM memory access patterns, but since the model files are stored on disk, you wouldn't ever keep them in memory anyway (except caching) and just reread them on demand.

➕ show 1 reply
vlovich123 • today at 4:13 PM

You are correct, compression doesn’t help reduce a model. That’s why quantization is inherently a lossy operation although there’s tricks to limit the perceived loss. That’s the only way to further compress the data.

One way to think about it is that if the weights were compressible, it’s a clear signal that better training could make the model smarter as there’s redundant information in the structure that could be used to store other meaning.

The other challenge with compressing weights of course is that you have to decompress them before you do math with them. Dequantizing weights is significantly cheaper and supports random access too (not that that matters too much but it helps in terms of leveraging SIMD techniques to do it in an embarrassingly parallel way)

➕ show 1 reply
matja • today at 12:45 PM

> A bit like trying to compress a JPEG in a zip file

Interesting comparison, I wonder if there are loss-less compression schemes that are "weight-aware", about the statistical properties of model weights.

Back to that line - because (Huffman coded) JPEG images can be lossless recoded using arithmetic coding, which makes them smaller than if they were compressed with the best general-purpose compression algorithms/tools (e.g. `jpegtran -arithmetic < input.jpg > output.jpg` beats `xz -9 < input.jpg > output.jpg`). AC takes advantage of the properties of the image coding better than general-purpose compression tools on the output of the Huffman coding.

➕ show 1 reply
rft • today at 11:11 AM

I haven't tried it in RAM, but your reasoning matches what I saw when I looked at compressing weights for archiving them. The weights don't compress nicely with standard, especially fast, algorithms. Some more background in this repo [1] including example compression ratios. I would also worry about decompression speed, that gives you a hard bound on the throughput. If the numbers in this repo are accurate, you likely hit the throughput limit of decompression way before you hit the memory limit, even considering the space savings and thus higher effective throughput.

[1] https://github.com/zipnn/zipnn#results

kalcode • today at 11:34 AM

I don't know at all. But I've been playing with oMlx and Splash. Using Pi there are compression extensions like the billion-context. I suspect your right.

But all seem to have massive issue boards. What I did see was oMlx release clustering so you can string 3 macs and have 3 times the memory or use one for prefilling accelerating.

Which I thought was neat and what I'll likely poke at instead of getting more ran.

➕ show 1 reply
strobe • today at 3:45 PM

it interesting because usually on workstation the browser & electron apps it's that holding much of the memory hostage (it's very common to see that browser consuming 20-30gb on 64gb system), so may be if it will be significant compression win on that side it leads to more memory for model but how much performance penalty would be for inference in that case?

➕ show 1 reply