logoalt Hacker News

jll29 • today at 9:00 AM • 3 replies • view on HN

Yes: you can classify a test file by topic with gzip as follows:

  gzip -9 sports.txt   testfile.txt

  gzip -9 politics.txt testfile.txt

  gzip -9 business.txt testfile.txt
(ass. sports.txt politics.txt and business.txt are text docs pertaining from the sports, politics and business domains, respectively, and have equal size)

The test file belongs to the topic with the smallest size *.gz file.

Witten's group at Waikato uni were perhaps the first to work on this.

Also check out the Hutter prize if you are interested in this.

Replies

stingraycharles • today at 10:35 AM

Back in the day - maybe two decades ago - I implemented language detection like this.

I seeded gzip compressors’ dictionaries with Wikipedia articles in different languages.

I would then try to use said dictionaries on any random text, and the one that was best able to compress it, was the correct language.

Absolutely totally not the best approach, but very fast and super simple to implement.

woadwarrior01 • today at 10:41 AM

aka Normalized compression distance (NCD). Its close cousin: Normalized Google distance (NGD) is also super interesting!

https://en.wikipedia.org/wiki/Normalized_compression_distanc...

LPisGood • today at 10:34 AM

There are some deep connections between machine learning, compression, and cryptography with information theory as a common thread.

Also, I’ve never seen “ass.” Used to shorten “aside” — I typically use N.B. but perhaps only for important ones.

➕ show 3 replies