logoalt Hacker News

kevin_nisbet • today at 4:26 AM • 1 reply • view on HN

I don't have the full story because it happened in another business unit, but one of the worst outages I ever worked the rumour was it was caused by someone doing prep work for a later cutover to new hardware. Whatever they were doing to prep for the upcoming cutover, the router rebooted. They didn't know it rebooted, so they did the same thing to the secondary and tertiary a few minutes later. Also happened to be on a Friday.

So sometimes it's easy to rehearse all the critical stuff, and get a bit complacent with the stuff that is just expected to work, plugging something in that's not configured and getting it ready for the big migration. And it's hard to get a 100% perfect replica in a lab. So I think odds are you're right, but I think there is also a longer tail of possibilities as well. And in my experience, there is always tension between chasing the 9's of availability, and getting some "safe" work done when convenient.


Replies

brookst • today at 5:31 AM

This is a fairly common pattern. I’ve also seen a few cases where a failover environment doubles as smoke test for prod rollouts. Hey, it’s a mirror of prod, and sits idle almost all the time, so it’s simpler and cheaper to test that way. Until prod goes down and a failover is triggered mid-deployment to the backup environment.