alt Hacker NewsSounds like a bad default Juniper setup for RTSP all or spanning tree max on. Which is not the factory default.
“FAA officials identified a feature on the router that appeared to be resulting in the data overload, according to government and industry officials. A government official described it as a broadcasting feature that sent data more broadly than necessary, and said the agency has since disabled it on all the Juniper routers.”
The engineers that write the guides for deploying routers and the technicians that field them could have an old or local version of install notes leading to this.
I’ve seen multiple versions of install guides floating around in emails with no documentation control leading to bad installs.
It reminds me of a time, over 20 years ago, when I was the main part of upgrading large plant's phone system to my company's on-premise VOIP offering.
It was more than a normal factory; it was a giant place that designed and built complex machines that were sent out to make other complex machines at much higher volume.
Anyway, the cutover was badly-planned but it went smooth-enough. I had issues with their PRI and the T1, and my fucking manufacturer rep who was supposed to be present on-site was completely fucking AWOL, but I got lasting help from some dude from Sprint who was exquisitely patient and we walked through switch configuration options at all levels, including for circuits that did not belong to Sprint.
It worked. It worked great, in fact. We met the deadline. There were some details to sort, but they weren't significant and everyone agreed that they could wait a day or three.
I dusted myself off after a 38-hour continuous shift, drove home, and collapsed on the sofa in the front room with a huge, bright south-facing window.
20 minutes later, my phone rings. It's my boss.
"Hey, uh. They say their network is completely hosed. For the entire plant. They want you to call them right now."
I know that all I did was replace a phone system and put in some VOIP desksets -- all I did was add some network endpoints -- but whatever. I call them.
It's positively fire and brimstone over there. "WHAT DID YOU DO TO OUR NETWORK! NOTHING WORKS! THE WHOLE THING IS JUST FULL OF PACKETS, AND NONE OF THEM FUCKING MEAN ANYTHING! PRODUCTION, OFFICE, EVERYTHING!"
yawn
I got my boots on, found some coffee, and started driving back.
When I walked back in I expected to be drawn, quartered, and gutted, but it was instead weirder than that: "WHY DID YOU PLUG BOTH PORTS OF THIS VOIP PHONE INTO THE WALL?"
I hadn't. None of us had. It was all working quite sensibly when we left.
"BOTH PORTS WERE PLUGGED INTO THE WALL! THIS CRASHED OUR ENTIRE NETWORK! WE LOOKED EVERYWHERE! AND WHEN WE UNPLUGGED YOUR PHONE, WITH BOTH OF ITS CABLES, THINGS GOT BETTER! WE'RE SO PISSED AT YOU RIGHT NOW!"
I was several layers beyond being too tired to care much about these strong words. And it was working again, so I left. I made it back home somehow.
---
What happened:
This plant had a switched network that they were very proud of. It used HP Procurve switches, in a special mesh mode. They were very pleased to explain, quite early on, exactly how robust this was: It was very fast when the mesh was intact, and it would adjust immediately to failures between switches and still be Fast Enough when the mesh was not intact.
But! This version of that topology was incompatible with STP, so STP was not turned on.
So when someone (not us-the-installers, nor them-the-plant-IT folks, but someone) plugged both dumb ports of the 10/100 switch in one lowly IP phone into that network: Their entire network shit the bed. Bad. It shit up the entire bed very completely.
(I understand that this HP Procurve mesh/STP incompatibility problem was subsequently resolved with firmware but that was well after this plant's network crash.)