Yes. And as someone who worked at Fog Creek at the time, that was a completely avoidable situation that should never have happened, and I'm absolutely gobsmacked we nearly had a repeat of it over a decade later.

Context: in 2012, I literally quit Fog Creek, largely over the fact that we had no backup strategy and I was told that a backup strategy was not feasible. I was "convinced" (paid a lot) to stay on for another two years to see if Kiln would succeed. On my way out of that meeting, I commented to a coworker, "I mean, what's the chance a hurricane hits Manhattan in the next 24 months?"

100%. The chance was 100%. And the only reason people had to haul open containers of diesel up 17 flights of stairs was because no one wanted to admit that the chance wasn't 0%.

Anyway, not sure what was going on with Jane Street here, but holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.

I was working at a data center in the financial district during Sandy. That was quite the experience. For whatever reason, our building was one of the last to have power turned back on. We never had to move diesel by hand fortunately. We did have a fun moment early on where one of our generators needed to have most of its load reduced for a few minutes. There were a pair of external fuel filters that needed to be changed fairly frequently. They were meant to be redundant so you could change one at a time. By the time we were able to get a tech on site, they were so dirty that when we started switching to just one the generator came very close to stalling out. After some conversation we thought it was thought that if we could drop the load enough we should be able to get away with just one dirty filter for a short period of time. The only chance we had at that was to shut down all of our cooling units. The generator was on the roof, and I was coordinating the timing over radio. You could dedinitely hear the generator wasn’t happy, but it worked. As soon as the second filter was replaced I let everyone know it was time to turn all of the cooling units back on. No one responded, and after a pause I realized that meant that probably no one had heard me and things were getting hot fast. I sprinted back inside. On the way my radio dropped, so once inside I sprinted to all of the units to tell people to turn them back on. For some reason I didn’t think about grabbing someone else’s radio. That may have been a lucky mistake though, as I’m not sure what all of those compressors turning at at the same time would have done.

I had made sure to observe the filter replacement process in case we needed to do it again ourselves. We did, and the next time someone recorded me doing it so everyone could learn and we handled most I’d the rest of the swaps from then on.

There were a lot of memorable experiences that week or so.

My colleague Brett has a Hurricane Sandy story about Jane Street as well

https://x.com/BrettHarrison/status/2117636364304892206

That one tested a lot of assumptions in planning for sure. I recall that we had a sound DR strategy, however it turns out, our main site was in Newyork and the DR site was in New Jersey to save costs and still be compliant. Good learning for sure.

Diesel is ~7lbs/gallon. That is truly superhuman damn.

I wonder how well these "superhuman folks" were paid

> The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.

Awesome, story time! I was working at a not brand new but still relatively small tech company (~$100m yearly revenue) and we were in two major datacenters, one in Las Vegas, and another in Manhattan. We were essentially telephony in the cloud, back when that was still fairly novel. At the time, most of our revenue came from handling various 1-800 campaigns, like "Ronco Knives" if you remember that, and also big money raising telethons. Our platform was active/active between the two datacenters. Of course not for a given phone call, but we could pretty easily balance traffic between both datacenters, even fully away from one of them, though as we'll see later, at some peril.

Our main networking guy was also the main datacenter person, and he'd been in NY to help set us up there. I can't remember which datacenter it was, but we had like half of a floor well above the street.

I was watching Sandy carefully, and once the odds of it hitting Manhattan were substantial, I started to press for plans to move all traffic out of the NY datacenter no less than 24 hours before landfall.

The network guy disagreed; he said that he'd been there, on site. He saw the redundant generators. The redundant network ingress and egress. The site people joked that if somehow the first floor ever flooded, the the rest of the datacenter would continue operating normally.

I noted that while all that might be true, we don't have any such assurances for all of the other equipment that'll be underwater needed to support the network.

I ended up escalating to our boss, the head of operations. He was one cube over. :) We had a vigorous discussion, and the VP agreed to pull our traffic out of NY. So we did.

Probably 75% of our capacity at the time was in LV and 25% in NY. We were running pretty hot utilization wise, but LV could, under normal circumstances, handle the whole platform by itself.

Sandy came and flooded. Generators kicked on. The network stayed up. Then the DC/networking guy got an emergency call from the datacenter.

They had plenty of fuel on site, but it turns out that the fuel inlets for the generators were...under water and could not be reached. So everything was going to go down in a couple of hours.

And it surely did, for reasons I've forgotten, in a very crappy way. Our logs indicated that power hard flapped for all of our equipment quite a few times.

We lost quite a few PSUs due to that. Yes, we definitely should have shut our equipment down cleanly. Oops.

Why the hell did they park the generators on the 17th floor where people have to hand carry diesel fuel is the real question.

The ars technica article says:

> “They had their brigade of people carrying five-gallon buckets up 18 flights of stairs or whatever it was to get to their auxiliary generator. Quite a task by those guys,” Orchard said.

5 gallons of fuel at a time (hopefully not in a literal bucket) is a lot easier than managing a big drum on stairs. Unless you have a powered stair dolly, but powering your stair dolly up 17 flights of stairs seems like a challenge too. Team lifting a drum up one flight of stairs feels managable, doing it up 17 flights multiple times is a hard no.

https://web.archive.org/web/20121103050116/http://searchdata...

Apparently it was a 30-person 5 gallon bucket brigade of customers, contractors, and volunteers all lining up to help fill tanks because of interconnection issues between the supplier they got and where they needed to fuel the tanks up with.

"... neither we nor the fuel company had the proper coupling to tie into that existing infrastructure."

Sometimes "DR" in your plans stands for "didn't realize".

As a bonus though? They were able to in fact get ahead of the fuel burn, and the crew managed an eight hours' head back up on the tanks.