> The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.

Awesome, story time! I was working at a not brand new but still relatively small tech company (~$100m yearly revenue) and we were in two major datacenters, one in Las Vegas, and another in Manhattan. We were essentially telephony in the cloud, back when that was still fairly novel. At the time, most of our revenue came from handling various 1-800 campaigns, like "Ronco Knives" if you remember that, and also big money raising telethons. Our platform was active/active between the two datacenters. Of course not for a given phone call, but we could pretty easily balance traffic between both datacenters, even fully away from one of them, though as we'll see later, at some peril.

Our main networking guy was also the main datacenter person, and he'd been in NY to help set us up there. I can't remember which datacenter it was, but we had like half of a floor well above the street.

I was watching Sandy carefully, and once the odds of it hitting Manhattan were substantial, I started to press for plans to move all traffic out of the NY datacenter no less than 24 hours before landfall.

The network guy disagreed; he said that he'd been there, on site. He saw the redundant generators. The redundant network ingress and egress. The site people joked that if somehow the first floor ever flooded, the the rest of the datacenter would continue operating normally.

I noted that while all that might be true, we don't have any such assurances for all of the other equipment that'll be underwater needed to support the network.

I ended up escalating to our boss, the head of operations. He was one cube over. :) We had a vigorous discussion, and the VP agreed to pull our traffic out of NY. So we did.

Probably 75% of our capacity at the time was in LV and 25% in NY. We were running pretty hot utilization wise, but LV could, under normal circumstances, handle the whole platform by itself.

Sandy came and flooded. Generators kicked on. The network stayed up. Then the DC/networking guy got an emergency call from the datacenter.

They had plenty of fuel on site, but it turns out that the fuel inlets for the generators were...under water and could not be reached. So everything was going to go down in a couple of hours.

And it surely did, for reasons I've forgotten, in a very crappy way. Our logs indicated that power hard flapped for all of our equipment quite a few times.

We lost quite a few PSUs due to that. Yes, we definitely should have shut our equipment down cleanly. Oops.