alt Hacker NewsThere's a great piece of software called "molly-guard", which intercepts calls to "poweroff" and "reboot" and similar. It checks if it's being invoked via an SSH session, and if so, it asks you to type the name of the system you're shutting down. That way, you never accidentally shut down a remote server when you meant to shut down your own system (or a different server).
Replies
magicalhippo • today at 5:33 AM
Another fun one is disabling the network interface on a remote server. An acquaintance did that by mistake on a cloud VM running some core services, and the cloud provider had no virtual console for some reason. Ended up having to write off the VM and restore from backup. Fun day at the office.
➕ show 3 replies
I once accidentally rebooted the reverse proxy for all our production traffic. We got some very quiet two minutes while it came back up.
After that we installed molly-guard with a check for the number of active connections. Made it painless to reboot standby proxies and difficult to reboot active ones.
(We also instituted pairing on production proxy maintenance. I'm not a fan of pair programming but pair maintenance is great.)
I like telling junior hires about this incident because it teaches them that (a) anyone can make mistakes, (b) even serious mistakes usually aren't that dangerous, (c) you can learn a lot from mistakes with the right mindset, (d) we cannot prevent mistakes but with the right system design we can reduce their consequences.