alt Hacker News> Tier 3 incidents: Novel, complex, cascading failures stay human-led, with AI providing supporting context.
> But here’s the catch.
> If human SREs (okay, now we have to use adjectives like human ) only engage with Tier 3 incidents, i.e. the novel, never-before-seen outages, where do they build the intuition to handle them?
How did they do this before AI? I have done a bunch of complex production level debugging and these Tier 3 incidents are always "starting from scratch" never seen before issues. The main "intuition" I built is persistence and verifying/refuting random hypothesis. In other words, I think we can build an intuition for Tier 3 issues only by engaging with Tier 3 issues.
Although, psychologically, some success in T1 and T2 issues is what gives us the confidence to even approach the T3 issues.
When you're just starting out in a system, every incident is a novel, never-before-seen issue. That's putting aside existing sources of knowledge like coworkers and runbooks of course, but I've had to do "software archeology" on lots of stuff because coworkers are gone and docs are wrong. But that's where you build up that intuition, like how the company typically builds things, how deployments usually work, what's that terrible load bearing service installed 10 years ago and not updated since, etc. Yeah every T3 incident eventually is novel, but being able to cut out the non-novel parts helps a lot