logoalt Hacker News

ssivarktoday at 8:28 PM4 repliesview on HN

Nope; there is a bit more nuance and the distinction is important.

Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems. The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support! Eg: you observe a rare edge case in your training data and (lossy) compression could simply ignore it. But if you wanted generalization in that particular part of the space -- either because an adversary was testing you, or for design freedom where you choose to build in that specific corner -- then you don't just want data compression, but good prediction performance on a test distribution which peaks in that corner.

Assuming that the training data distribution is exactly the distribution you will ever care for is implicitly doing a lot of the heavy lifting in the claim that compression = prediction, and I'm peeved at how much this statement is unthinkingly repeated like a manifesto.

There is nothing natural about the training data distribution, especially if the data generation process is exploratory while the downstream usage will be exploitative.


Replies

porphyratoday at 9:11 PM

How does that invalidate the "compression is prediction"? If the future data is different and you failed to generalize, then the failure to predict means you got worse at compressing and have to spend more bits storing the new information. Conversely, if the future data is the same as that you've seen previously, you could predict it very well, and compress better as a result.

show 3 replies
variadixtoday at 9:52 PM

I’m not exactly sure what distinction you are trying to make, but a compressor that generalizes from currently observed day to potential future data _is_ a better compressor, as it will have better predictions as it observes more of the source data distribution. Maybe you are making a point about whether compressors can “learn” after compressing the input data, but this also isn’t a fundamental property of compressors. Also compressors do have to deal with non-stationary data sources, so it also isn’t correct to say a compressor only models stationary distributions.

jbs789today at 8:57 PM

That’s interesting.

Also sparked the thought that the assumption only holds if the future looks like the present.

show 1 reply
schopra909today at 8:56 PM

100% agreed.