Making 768 servers look like 1

i do wonder how something like this can be generally implemented. i presume this must only support a subset of SQL/plpgsql, as some things would be.. utterly insane to manage manually. e.g., if i have a table with a btree-gist overlap constraint, or some inclusion-exclusion check-constraint (or literally any constraint that requires multiple rows to be fully determined - there are quite a lot of them), how on earth does this work?

there's a reason why postgres writing is (mostly) serialised (asterisk) to a single writer (asterisk asterisk). something something ACID, but in short by having multiple writers improves availability, but weakens integrity.

I disagree with the opening premise:

> A single database server cannot handle such demand, so we must spread the queries and data out across many servers with database sharding

Did you max out the capacity of the best server you can buy?

Such a database can serve millions of customers (the numbers given).

You always want to scale up the other parts first, request handlers, caching, etc. The day you can no longer inspect the essential state of your system is the day your company better be included in NASDAQ and ready to pay a few hundred engineers 300k salaries.

Took me an embarrassing amount of time to realize this is an ad.

What about sequences? The example shows an auto-incrementing user ID. How’s that possible without contention between all shards? Is the proxy responsible for sequences?

What about foreign keys? Do they all have to live on the same shard? How do you do distributed transactions?

On cross-shard reads: how do you do sorting? And cross-shard joins?

I’d love to be proven wrong, but I suspect the 768 servers look like 1 only on the very surface, and you’ll get wildly different characteristics from cross-shard and single-shard queries.

I personally would prefer if they _didn’t_ look like 1 if they can’t behave like 1.

Sibling post has author answering questions in comments: https://news.ycombinator.com/item?id=48925420

Looks like the GIF is fully built out in code. It's really nice to look at, well made, and easy to understand too. I wonder what program or code they used. I'd love to know.

p.sI thought it was a GIF, but it's an iframe. That was a nice little surprise.

Load balancers, microservices and horizontal scaling?

Previously: https://news.ycombinator.com/item?id=48925420