logoalt Hacker News

paulryanrogerslast Saturday at 6:25 PM7 repliesview on HN

Disks go bad. RAID is nontrivial to set up. Hetzner had a big DC outage that lead to data loss.

Off site backups or replication would help, though not always trivial to fail over.


Replies

alemaneklast Saturday at 6:48 PM

As someone who has set this up while not being a DBA or sysadmin.

Replication and backups really aren’t that difficult to setup properly with something like Postgres. You can also expose metrics around this to setup alerting if replication lag goes beyond a threshold you set or a backup didn’t complete. You do need to periodically test your backups but that is also good practice.

I am not saying something like RDS doesn’t have value but you are paying a huge premium for it. Once you get to more steady state owning your database totally makes sense. A cluster of $10-20 VPSes with NVMe drives can get really good performance and will take you a lot farther than you might expect.

show 3 replies
fabian2klast Saturday at 8:33 PM

For this kind of small scale setup, a reasonable backup strategy is all you need for that. The one critical part is that you actually verify your backups are done and work.

Hardware doesn't fail that often. A single server will easily run many years without any issues, if you are not unlucky. And many smaller setups can tolerate the downtime to rent a new server or VM and restore from backup.

mcnyyesterday at 10:57 AM

One thing that will always stick in my mind is one time I worked at a national Internet service provider.

The log disk was full or something. That's not the shameful part though. What followed is a mass email saying everyone needs to update their connection string from bla bla bla 1 dot foo dot bar to bla bla bla 2 dot foo dot bar

This was inexcusable to me. I mean this is an Internet service provider. If we can't even figure out DNS, we should shut down the whole business and go home.

PunchyHamsteryesterday at 12:59 PM

They, do, it isn't, cloud providers also go bad.

> Off site backups or replication would help, though not always trivial to fail over.

You want those regardless of where you host

mattmanserlast Saturday at 7:50 PM

So can the cloud, and cloud has had more major outages in the last 3 months than I've seen on self-hosted in 20 years.

Deploys these days take minutes so what's the problem if a disk does go bad? You lose at most a day of data if you go with the 'standard' overnight backups, and if it's mission critical, you will have already set up replicas, which again is pretty trivial and only slightly more complicated than doing it on cloud hosts.

j45last Saturday at 7:39 PM

Not as often as you might think. Hardware doesn’t fail like it used to.

Hardware also monitors itself reasonably well because the hosting providers use it.

It’s trivial to run a mirrored containers on two separate proxmox nodes because hosting providers use the same kind of stuff.

Offsite backups and replication? Also point and click and trivial with tools like Proxmox.

RAID is actually trivial to setup.l if you don’t compare it to doing it manually yourself from the command line. Again, tools like Proxmox make it point and click and 5 minutes of watching from YouTube.

If you want to find a solution our brain will find it. If we don’t we can find reasons not to.

show 1 reply
znpyyesterday at 4:43 PM

> RAID is nontrivial to set up.

Skill issue?

It's not 2003, modern volume-managing filesystems (eg:ZFS) make creating and managing RAID trivial.