alt Hacker NewsCompression can be defined as reducing uncertainty. If you can predict the next sequence you can compress it to 0 bytes using arithmetic coding. Reliable prediction is what enables compression and it's the link between compression and AI that everyone is talking about.
No one ever in comp sci says artificial intelligence is "like compression", they correctly state that "artificial intelligence IS compression". It's absolutely known and accepted that artificial intelligence (defined as predicting outcomes with a measure of certainty and taking chosen actions towards goals using those predictions) has equivalence to compression in a very hard science way. The hardest part of artificial intelligence is compression and the remaining part, the choice of actions based on predictions is just a tree search to a goal.
Replies
The factor that is missing in that analysis to me is a time based dynamic stability perspective. Humans have a pretty good ability to go off the rails in reasoning one day and wake up reasonable; a pretty good ability to pursue tasks, despite a multitude of distractions, for ten years or longer. The best models get appreciably worse over a half million tokens. Even using a bunch of limited context agents over time, they lack mental stability. They keep coming up with ideas contrary to the long term idea, and every so often generate ideas that make no sense but they have a hard time letting go of. So the pure functional LLM is compression, but AGI needs some centering process, some high level of dynamic stability to stay sane over time and in the face of 10,000 shiny pretty things to chase.
The harnesses get better, but I haven’t seen much experimentation on long term stability, at least since the “let the LLM run the candy machine” papers from a while ago.
Because the thing missing, even with the largest agentic swarms, is independent intelligence, where it’s given something to own, like say “end to end data quality as we add more clients” (for a SaaS) and it just figures out what that means at each time, mutating its role and solutions to fix the external world, without getting silly.
Compression in image, video, sound, and text. These items to compressed are all created by humans and we will say represented by files. The difference between an instant of reality and the files is vast. Reality also doesn’t stand still and each instant needs to be captured and interpreted before AI happens.
AI can be just like compression but currently the compute power is no match for details.
Finally these reality details need consideration in any successful implementation. Which means the implementator needs to be aware of the details and successfully relate them to everything else in the model.
I think anyone surprised by these things is not fully engaged with what they are doing.