Viewing LLMS as compression is flawed because it shouldn't be possible to solve new novel problems. Zipping every single file in existence and then asking Winzip to produce a novel working zip file clearly doesn't work. Clearly LLMs are learning from data it is fed and a lot of what we do before LLMs also involved "plagirism". Even the greatest writers and artists would copy each other to master their styles and extract the dimensions. I think framing LLMs as compression is not only silly it demonstrates a poor understanding of the inner workings of LLMs
Except a statistically salient search context is in proximity to something similar, and manual content curators drop their thumb on the scales weights to boost desirable outcomes in the quantized model.
https://www.youtube.com/watch?v=LPZh9BOjkQs&list=PLZHQObOWTQ...
The encoding of the LLM is almost always lossy compression, but we agree it is not real intelligence.
As a sentient turnip, I likely wouldn't understand such things, but do recognize copyright bleed-through when it includes known obscure works. =3
https://en.wikipedia.org/wiki/Mechanical_Turk