alt Hacker NewsThe Single Router in Southern Arkansas That Ground Air Travel to a Halt
Comments
Sounds like a bad default Juniper setup for RTSP all or spanning tree max on. Which is not the factory default.
“FAA officials identified a feature on the router that appeared to be resulting in the data overload, according to government and industry officials. A government official described it as a broadcasting feature that sent data more broadly than necessary, and said the agency has since disabled it on all the Juniper routers.”
The engineers that write the guides for deploying routers and the technicians that field them could have an old or local version of install notes leading to this.
I’ve seen multiple versions of install guides floating around in emails with no documentation control leading to bad installs.
This router should be on display in the Smithsonian air and space museum in dc. It belongs in a museum!
It would be nice to know what was disabled on the juniper routers.
For the network engineers on here: does this explanation make sense to you?
all this ip-based shit is incredibly complex. the fact that facebook, google, etc have all had huge outages related to bgp shows that even the highest paid and most talented network engineers will make mistakes. now i think L3Harris is still incompetent with their design and safeguards, and thats probably why we dont even have a proper postmortem here. but the amount of complexity and failure modes is unarguably higher after these upgrades.
So why didn't they replace the cables and router SW overnight? There's not much air traffic at night, it's really trivial compared to all other cases. Amateurs. And then they are asking for $50b.
So... It was BGP and not DNS this time?
Also, "the flood of messages" sounds suspiciously like a routing loop somewhere.
"See this router here? If I switch it off, the ENTIRE US AIRSPACE GOES DOWN."
"Sure thing buddy, I believe you."
The irony of not being able to read the article because archive.is giving a 500 error for the page
so everything we were told was correct:
1. it was L3Harris (CBS news): check
2. it was a "newer system overloading the legacy system" (Duffy): check
3. it was "old copper wires" (Duffy): check
4. it was not related whatsoever to the SMART AI system: check
5. the process of upgrading the legacy systems is part of the problem: check
6. the combination of a "newer" system with an "older" system, something that likely should have been planned for if the contractors weren't being rushed, was involved [1]: check
[1] https://www.flyingpenguin.com/faa-outage-why-duffy-is-spelle... - this is a much better article and no paywall
A buddy was asking me if this story "holds water" earlier today (because I'm more into networking). I eventually decided it was plausible, but only through at least one, and probably multiple "rookie mistakes".
As described it was in the course of switching out copper TDM lines with fiber. On it's face that could be straightforward, but it's easy to imagine the router(s) terminating these lines didn't support the required line cards, so router upgrade(s) were required. So theoretically straightforward could have become more involved.
However, when doing far less critical updates, it's common to set up a test environment, do a dry run of the work, test the roll-back, test the monitoring and alerting, that sort of thing. Doesn't guarantee no outages, and I'm playing armchair QB here, but it seems like a botched job.
It's also telling the Transportation Secretary said things like "we aren't sure they if they aren't doing a great work for us". He also said "this is something businesses have been doing for decades and the FAA is the last to do it"; you'd assume that this is a process that someone (maybe not those doing the work) have developed deep experience with.
I'm not familiar with this L3HARRIS, I gather they are a defense contractor with billion dollar revenue.
Lots of things in this definitely raised my eyebrows.