alt Hacker News> "Just as the FAA mandates manual flying time even when the autopilot is perfectly capable, SRE teams can designate certain incidents, even the ones the AI could handle, as human-practice opportunities"
This is a great example. Aviation is a perfect industry to learn from, they figured out a lot of things about how to keep humans from making mistakes. While complexity increases and automation takes away more and more tasks.
Replies
A (very) key difference is that this behaviour is mandated by the FAA, and consistent across all providers. There isn't a (legal) option of just not-doing-it because it's "cheaper".
Modern companies generally do not have the same incentive to keep their SRE (or engineers in general) upskilled. Cutting SRE because "we don't really have incidents, why do we keep paying them? If we need to do layoffs I would rather keep a product engineer" will always* (okay, _generally_) be the direction that organizations go without external influence or requirements.
Now I am sure there are some companies or industries where this is required (for example, I would not be surprised if certain military software/services contracts had requirements for support staff with X hours or training/etc), but that is not consistent everywhere.
If we want something that is "money-losing" in the short/medium term to be consistently applied, it unfortunately needs to be (uniformly) regulated.
The thing is, without the FAA mandating something like this for SRE’s, no one will do it. In the moment, getting the broken thing back up and running is seen as far more important than the learning opportunity.
I noticed this constraint a few years ago when the SRE/DevOps hiring market was really tight, but no one was investing in non-senior talent. I ended up founding a startup that tried to address this by supplying training via simulated outages. Maybe we were too early.
This was just the default where I worked. We called the exercise Wheel of Misfortune: facilitator sets up an outage / bug in a non-prod environment (inspired by real events or not), spins a wheel, picks a victim, and the problem is solved collectively, manually. That's the standard format, but it could be much more freeform than that: the point is to educate each other regarding "incident management."
I will say, I'd always been skeptical that AI could be better at incident management than an experienced engineer with deep domain / institutional knowledge, but I was recently proven wrong: Claude root-caused and helped resolve a Postgres lock queue pileup that I believe would have taken me much longer to fix.
Re: aviation being a perfect industry to learn from, I think https://m.youtube.com/watch?v=5ESJH1NLMLs might be a good watch. I was introduced to this at a Wheel of Misfortune around 5 years ago, so it's interesting watching it today when AI is encroaching upon everything.
Just like pilots, SREs will have to spend a minimum number of annual hours in the simulator to keep their skills sharp.
Aviation is a perfect industry to learn from if you are willing to hold IT to the same standard of regulatory scrutiny. You can bet your entire net worth that anyone running anything in the industry would fight tooth and nail to prevent that sort of regime from coming into force.
> Aviation is a perfect industry to learn from, they figured out a lot of things about how to keep humans from making mistakes.
Aviation is used to paying through the nose for stuff. For developers, employers want to pay as little money as possible.
Fuck aviation, stop using it as a dead horse metaphor for safety through human protocols. I am private instrument rating holder and there's very much a top down push for single pilot ops. Your newer jets (even the jetliners) are all designed with that in mind even though they may not be certified as such.
Aviation is different because their rent-seeking is enforced by the required legal hedging. I'm not sure the same incentive structure exists in the software side of things.
In other words, we might need to find some hobbies.
Aviation is a good example if you want to have zero fatal mistakes. There is a way and there is a cost to do so.
The same way erlang is good example of how to run a highly available system with hot reload and observability and stuff. There is cost for that too.
Maybe not all of the projects need the same properties. I'm fine if some of the stuff is having scheduled or unscheduled maintenance. Some kind of government portal that I use once a year had my permission to be offline between 0300 and 0500 and then again after 1200.