alt Hacker NewsThe cost of using a textual format is that floats become so slow to parse, that it’s a factor of over 14 times slower than parsing a normal integer. Even with the fastest simd algos we have right now.
Replies
HelloNurse • today at 10:08 AM
So it depends. Float parsing performance is only a problem if you parse many floats, and lazy access might reduce work significantly (or add overhead: it depends).
➕ show 1 reply
meehai • today at 8:49 AM
and with little data (i.e. <10Mb), this matters much less than accessibility and easy understanding of the data using a simple text editor or jq in the terminal + some filters.
➕ show 1 reply
if you data is lots and lots of arrays of floats, this is likely not the format for you. Use float arrays.
Also note it stores decimal in a very compact encoding (two varints for base and power of 10)
That said, while this is a text format, it is also technically binary safe and could be extended with a new type tag to contain binary data if desired.