logoalt Hacker News

Fc, a lossless compressor for floating-point streams

26 pointsby endukulast Sunday at 10:14 AM7 commentsview on HN

Comments

loegtoday at 2:21 AM

The question is, how close can OpenLZ come? (This is from the same people who develop zstd, but suitable for structured data in a generic way.)

Scaevolustoday at 1:07 AM

I see you have ALP, but have you tried Chimp128 or Arrow's byte stream split?

KerrickStaleytoday at 1:51 AM

Another library in this space is pcodec; I'd appreciate a comparison of the two.

endukulast Sunday at 10:14 AM

I built "fc", a C library for compressing streams of 64-bit floating-point values without quantization.

It is not trying to replace zstd or lz4. The idea is narrower: take blocks of doubles, try a set of float-specific predictors/transforms/coders, and emit whichever representation is smallest for that block.

It is aimed at time-series, scientific, simulation, and analytics data where the numbers often have structure: smooth curves, repeated values, fixed increments, periodic signals, predictable deltas, or low-entropy mantissas.

The API is intentionally small: "fc_enc", "fc_dec", a config struct, and a few counters to inspect which modes won. Decode is parallel and meant to be fast; encode spends more CPU searching for a better representation.

Current caveats: x86-64 only for now, tuned for IEEE-754 doubles, research-grade rather than production-hardened.

Repo: https://github.com/xtellect/fc

show 2 replies