logoalt Hacker News

thaumaturgy • yesterday at 10:47 PM • 1 reply • view on HN

Snark aside, this is actual advice in Crisis Engineering by Nitze, Weaver, and Dickerson, and largely matches my experiences as well. You can harp on and on and on at upper management about why budget or attention needs to be allocated to take care of something before it blows up into a much bigger problem, but they'll never move on it -- until it actually does blow up, and then suddenly you've got nearly a blank check to do whatever's necessary to make sure it never happens again.

If you don't take full advantage of that, you're missing out on possibly your only opportunity ever to fix whatever's needed fixing for years.


Replies

sfink • yesterday at 11:53 PM

That makes sense, and yet...

If things blow up because you're upgrading to something supposedly better and it goes wrong, then that's not a time when you can properly fix the new thing. You can only push forward with the transition or go back to the old. I disagree with "you've got nearly a blank check to do whatever's necessary to make sure it never happens again." You have a blank check, but anything you use it for has to be scoped within stopping the bleeding. As soon as you're onto the new or back to the old, the check expires and you likely can't touch anything real anymore because it'll be involved in lawsuits and finger-pointing.

I don't disagree with the core message, but I think it plays out differently for a major upgrade vs a production meltdown on an existing system. With mostly an existing system, you can pour effort into resilience. With an upgrade, I doubt people would let you beef up a system that is still on the chopping block, even if it's getting a life extension. And if you're still switching to the new system, all effort is going to go into getting it up and running and anything that gets in the way of that (as quality improvements are likely to) is going to face some serious resistance. The check is not truly blank.