logoalt Hacker News

jasodetoday at 11:13 AM1 replyview on HN

>Looks very much like a format that should just have been gzipped JSON.

For application file formats that require storing binary blob data such as images, bitmaps, etc , many in the industry have settled on "SQLite db as a file format": (https://www.sqlite.org/appfileformat.html)

Examples include Mozilla Firefox using sqlite db for favicons, Apple iOS using sqlite to store camera photos, Kodi media player uses sqlite for binary metadata, Microsoft Visual C++ IDE stores source code browsing data in sqlite, etc.

Sqlite db would usually be a better choice rather than binary blobs encoded as Base64 and being stuffed into json.gzip files. One of the areas where the less efficient gzipped JSON might be better than Sqlite is web-server-to-web-browser data transfers because the client's Javascript engine has builtin gzip decompress and JSON manipulation functions.


Replies

OskarStoday at 2:10 PM

My favorite example of this is Audacity, which stores their entire project file, including all the audio data, as a SQLite database. It's really amazing, you can stream audio data from many input sources into a SQLite database at once, it wont break a sweat, it's flexible and extremely safe from data loss due to crashes or power outages. As well as trivially mutable: many of these kinds of formats the "export" is a heavy-duty operation which serializes everything to JSON or whatever. But in SQLite, it's just a question of updating the database with normal INSERTs, very cheap and simple operations. We've put a similar system into production, and it's incredible how well it works.