alt Hacker NewsThose care about quickly sending compact messages over the network, but most of them do not create a sparse in-memory representation that you can read on the fly. Especially in javascript.
This lib keeps the compact representation at runtime and lets you read it without putting all the entities on the heap.
Cool!
Amazon Ion has some support for this - items are length-prefixed so you can skip over them easily.
It falls down if you have e.g. an array of 1 million small items, because you still need to skip over 999999 items to get to the last one. It looks like RX adds some support for indexes to improve that.
I was in this situation where we needed to sparsely read huge JSON files. In the end we just switched to SQLite which handles all that perfectly. I'd probably still use it over RX, even though there's a somewhat awkward impedance mismatch between SQL and structs.