logoalt Hacker News

mrkeen • today at 3:07 PM • 0 replies • view on HN

I'm skeptical in the opposite direction (i.e. too difficult to have been solved)

I believe there's a tension between high-throughput (distributed) and centralised (id++), and I'm not sure it can be wished away.

I built a paxos implementation that could assign values to sequential keys. It runs at a decent speed when there's no contention, but slows down when different paxos nodes are asked to assign to the same key (they argue and retry etc.). A partial solution is to use a leader (i.e. putting id++ on one node), which prevents that kind of chatter.

Another partial solution is topics/partitions (which is just a fancy way of keeping independent id++ counters that don't interfere with one another.). I don't count this as a solution because it's morally equivalent to running multiple clusters that don't know about each other.

I can't quite picture the bookkeeping that would let 10 nodes record contiguous ids without paying the centralisation cost.