alt Hacker NewsYou basically can't do row by row appends to any columnar format stored in a single file. You could kludge around it by allocating arenas inside the file but that's still a huge write amplification, instead of writing a row in a single block you'd have to write a block per column.
Replies
gregw2 • today at 12:24 AM
Agreed.
There is room still for an open source HTAP storage format to be designed and built. :-)
You can do row by row appends to a Feather (Arrow IPC — the naming is confusing). It works fine. The main problem is that the per-append overhead is kind of silly — it costs over 300 bytes (IIRC) per append.
I wish there was an industry standard format, schema-compatible with Parquet, that was actually optimized for this use case.