logoalt Hacker News

curt15yesterday at 9:50 PM2 repliesview on HN

How does this relate to the work presented a few years ago by the ZFS devs using S3 as object storage? https://youtu.be/opW9KhjOQ3Q?si=CgrYi0P4q9gz-2Mq


Replies

magicalhippoyesterday at 10:54 PM

Just going by the submitted article, it seems very similar in what it achieves, but seems to be implemented slightly differently. As I recall the DelphiX solution did not use a character device to communicate with the user-space S3 service, and it relied on a local NVMe backed write cache to make 16kB blocks performant by coalescing them into large objects (10 MB IIRC).

This solution instead seems to rely on using 1MB blocks and store those directly as objects, alleviating the intermediate caching and indirection layer. Larger number of objects but less local overhead.

DelphiX's rationale for 16 kB blocks was that their primary use-case was PostgreSQL database storage. I presume this is geared for other workloads.

And, importantly since we're on HN, DelphiX's user-space service was written in Rust as I recall it, this uses Go.

tw04yesterday at 10:47 PM

AFAIK it was never released, and it used FUSE, it wasn’t native.