alt Hacker NewsGarage – An S3 object store so reliable you can run it outside datacenters
Comments
Looks interesting for something like local development. I don't intend to run production object storage myself, but some of the stuff in the guide to the production setup (https://garagehq.deuxfleurs.fr/documentation/cookbook/real-w...) would scare me a bit:
> For the metadata storage, Garage does not do checksumming and integrity verification on its own, so it is better to use a robust filesystem such as BTRFS or ZFS. Users have reported that when using the LMDB database engine (the default), database files have a tendency of becoming corrupted after an unclean shutdown (e.g. a power outage), so you should take regular snapshots to be able to recover from such a situation.
It seems like you can also use SQLite, but a default database that isn't robust against power failure or crashes seems suprising to me.
BTW https://deuxfleurs.fr/ is one of the most beautiful website I have ever seen
Seeing a ton of adoption of this after the Minio debacle
https://www.repoflow.io/blog/benchmarking-self-hosted-s3-com... was useful.
RustFS also looks interesting but for entirely non-technical reasons we had to exclude it.
Anyone have any advice for swapping this in for Minio?
I tried it recently. Uploaded around 300 documents (1GB) and then went to delete them. Maybe my client was buggy, because the S3 service inside the container crashed and couldn't recover - I had to restart it. It's a really cool project, but I wouldn't really call it "reliable" from my experience.
No tags on objects.
Garage looks really nice: I've evaluated it with test code and benchmarks and it looks like a winner. Also, very straightforward deployment (self contained executable) and good docs.
But no tags on objects is a pretty big gap, and I had to shelve it. If Garage folk see this: please think on this. You obviously have the talent to make a killer application, but tags are table stakes in the "cloud" API world.
I love garage. I think it has applications beyond the standard self host s3 alternative.
It's a really cool system for hyper converged architecture where storage requests can pull data from the local machine and only hit the network when needed.
No erasure coding seems like a pretty big loss in terms of how much resources do you need to get good resiliency & efficiency
One really useful usecase for Garage for me has been data engineering scripts. I can just use the S3 integration that every tool has to dump to garage and then I can more easily scale up to cloud later.
How is garage for a simple local dev env ? I recently used seaweedfs since they have a super simple minimal setup compared to garage which seemed to require a config file just to get started.
Wasn't expecting to see it hosted on forgejo. Kind of a breath of fresh air to be honest.
I use Syncthing a lot. Is Garage only really useful if you specifically want to expose an S3 drop in compatible API, or does it also provide other benefits over syncthing?
Corrupts data on power loss according to their own docs. Like what you get outside of data centers. Not reliable then.
Is it the same consistency model as S3? I couldn't see anything about it in their docs.
Anybody understand how this compares with Vast?
Unfortunately, this doesn’t support conditional writes through if-match and if-none-match [0] and thus is not compatible with ZeroFS [1].
https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main-...
this is the reliability question no?
Copy/paste from a previous thread [0]:
We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy in comparison to our existing use of MinIO, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s, over 10 nodes)
I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case.
---
In addition:
Next time we come to this we are going to look at RustFS [1], as well as Ceph/Rook [2].
We can see we're going to have to move away from MinIO in the foreseeable future. My hope is that the alternatives get a boost of interest given the direction MinIO is now taking.
[0]: https://news.ycombinator.com/item?id=46140342
[1]: https://rustfs.com/
[2]: https://rook.io/