logoalt Hacker News

roncohen • today at 10:35 AM • 2 replies • view on HN

As someone who myself worked on a hobby-level Rust based Kafka alternative that used Raft for metadata coordination for ~8 months: nice work!

Wasn't immediately clear to me if the data-plane level replication also happens through Raft or something home-rolled? Getting consistency and reliability right with something home-rolled is challenging.

Notes:

- Would love to see it in an S3-backed mode, either entirely diskless like WarpStream or as tiered storage.

- Love the simplified API. If possible, adding a Kafka compatible API interface is probably worth it to connect to the broader ecosystem.

Best of luck!

Replies

seanhunter • today at 11:20 AM

It says on the github page

   " It provides fault-tolerant streaming with automatic leadership rotation, segment-based partitioning, and Raft consensus for metadata coordination."
So I guess that's a "yes" to raft?
➕ show 1 reply
nubskr • today at 12:21 PM

Hi, the creator here, I think its a good idea to have S3 backed storage mode, its kinda tricky to do it for the 'active' block which we are currently writing to, but totally doable for historical data.

Also about the kafka API, I tried to implement that earlier, I had a sort of `translation` layer for that earlier, but it gets pretty complicated to maintain that because kafka is offset based, while walrus is message based.

➕ show 1 reply