logoalt Hacker News

Distributed, high-performance blob storage on SQLite

138 points • by ghostfoxgod • today at 8:39 AM • 26 comments • view on HN

Comments

anacrolix • today at 12:05 PM

I spent years trying to squeeze this use case out of sqlite. All roads lead to tears, it is not made for this use case.

I wrote https://github.com/anacrolix/possum for this exact use case. It has a few rough edges but it absolutely hauls ass for caching on disk. Incidentally it uses sqlite for transaction/TTL metadata which is more suited to sqlite but it's still the bottleneck.

sqlite transaction commit overhead is too high and uncompetitive without batching but it's a design flaw specific to being multiprocess.

➕ show 1 reply
lukax • today at 1:45 PM

OpenStack Swift uses SQLite for storing metadata. Each account has one SQLite db. Each container also initially had one SQLite db but in 2018 they added support for sharding so if a container has more than 1 million objects, the container is split into multiple SQLite databases. The performance is not ideal. Ceph scales much better.

gkamal • today at 1:23 PM

This looks like a classic case of nih/reinventing wheel.

bob1029 • today at 10:33 AM

> JSON blobs

An alternative to faster blob storage is to represent the blob's schema in SQL directly.

If most blob updates are incremental you can eliminate a significant amount of storage IO and garbage collection this way.

One SQLite database per blob type with a metadata SQLite db that tracks each. We could leverage powerful AI reasoning models to produce SQLite schemas and virtual table adapters for each encountered JSON schema type.

matja • today at 11:40 AM

What is the consistency model? What happens if a node is down unexpectedly?

➕ show 1 reply
CyrusGrissom • today at 10:25 AM

I couldn't find the word "replication" in the post or on the GitHub page. So, what about fault tolerance?

➕ show 1 reply
Blobtits • today at 1:11 PM

Reads like this is be a perfect fit for ClickHouse. You should do async inserts, remove the 20 or so _log options ClickHouse enables for some insane defaults and batch them in Redis if needed (probablynot). Then you have automatic compression, free ttl, no write concurrency.

feverzsj • today at 9:36 AM

Sounds like you should use a database-per-user design.

➕ show 1 reply
Kalanos • today at 11:47 AM

A limitation that i ran into with sqlite was not being able to interact with it over the network e.g. NFS. Does this solve that problem?

martinjc • today at 11:52 AM

Why not rqlite?

➕ show 1 reply
murkt • today at 12:01 PM

Sounds like it solves their particular problem, but hardly can be used by outsiders in the current state.

Bad fault tolerance, no words on replication or backup. Offline rebalance... meh. I am not sure how I feel about their choice of Redis protocol.

zodwick • today at 10:42 AM

all roads lead to sqlite

robinduckett • today at 11:25 AM

What about using, ahem, files?

➕ show 1 reply
Kalanos • today at 11:48 AM

Is the db still a single, portable file?

ratherlongname • today at 12:48 PM

So many comments about things already addressed in the article sigh

_s_a_m_ • today at 11:40 AM

so it's just sharding?

agline_in • today at 2:30 PM

[flagged]

wangguode • today at 1:54 PM

[flagged]

patryk_builds • today at 12:52 PM

[flagged]

aman179102 • today at 12:06 PM

[flagged]

foolgry • today at 11:12 AM

[flagged]

vrox • today at 9:28 AM

[flagged]

mesm7903 • today at 9:47 AM

[flagged]

MalgorzataKowal • today at 11:15 AM

[flagged]