logoalt Hacker News

8organicbitstoday at 3:34 PM4 repliesview on HN

I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3.

I ended up using gzip because it's best supported by the software I use and most likely to have support in software I adopt. But it gave the worst compression results of the options I tried. These bzip3 numbers certainly give me FOMO...


Replies

nvme0n1p1today at 3:53 PM

zstd is the go-to compression format these days. It's even supported in low-level software such as many linux filesystems.

I don't know much about duckdb but it looks like it supports zstd too: https://duckdb.org/docs/lts/data/json/loading_json

show 4 replies
bob1029today at 4:48 PM

For structured logs and json I've found a lot of success with PPM-style schemes.

If your JSON file has many of the same object, you could see ratios in the single digits.

woadwarrior01today at 4:42 PM

I'd recommend trying openzl for jsonl.

Danochtoday at 4:33 PM

[dead]