alt Hacker NewsSRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
Replies
Google's SRE was always about building infrastructure to make things reliable, and started in a world where LAMP on bare-metal + DNS load balancing was common. It makes sense to have a separate org/team/role responsible building real load balancing with health checks and failover, figuring out how to do zero-downtime updates, etc.
When these problems because largely solved with off-the-shelf software SRE became either a bureaucracy (ensuring reliability through policy and enforcement) or just deploying other people's code and holding the pager.
I need the company to have skin in the game, so I prefer that I am entirely unreachable outside M-F 9-5. Otherwise they just exploit people who submit far too quickly to being on-call 24/7. It also gives me motivation to make my stuff work as best as possible so that I'm not being annoyed by management about my non-existence outside core hours.
But I really do need the company to have a competent plan for handling issues that isn't just, "call the developer." It's amazing how much it makes them care about proper support and testing when the alternative isn't exploiting a developer, but is a customer outage.
> Real ones are oncall for the systems they build.
Yes! This is devops, thank you for coming to my ted talk.
Coincidentally I’ve found putting engineers oncall for their stuff really does wonders to ensure they check their AI agents’ code and know how it works.
On-call problems are mostly just a result of Kubernetes' "cattle not pets" brainrot ideology where people throw local simulate-ability out the window because the Rube-Goldberg web of pods and operators and cluster state gets too complex to simulate locally or keep in your head (esp. if code is located in 50+ microservice repos).
Machines (esp. Virtual Machines) are ultimately predictable and crashes don't "just randomly happen". But it's what this entire "enterprise cloud" culture has brought in, the idea that it's impossible to simulate all possible failure conditions of a cluster locally and therefore we shouldn't even try doing it. Maybe that's true at Google scale, but most companies are simply not, not, not Google-scale.
As a counter-example, I recently built a simple "cloud storage" system with a minimal Linux ISO: everything is stripped down to the absolute minimum to just boot, start a minimal S3 server and pull (encrypted and signed) update commands from an SQL "control plane" database. The entire userspace is ~50k lines of Rust (minus 10k init system) and I can simulate the entire VM locally and play with the "cluster" of VMs locally. The ISO is ~20 megabytes large and I can simulate entire datacenters "burning down", virtually, locally, before it happens for real. I can simulate everything: network failures, disk failures, slowdowns, latency, try getting Fable to hack it, etc.
But in the kubernetes "enterprise" world, I can't do any of this. First I need 10.000 values.yaml files, 10 add-ons for logging and metrics (because pushing metrics to some collection endpoint over HTTP is clearly PhD level complexity) and on top you need a service mesh to restrict random containers talking to each other because nobody knows anymore what services can talk to which other services. The results are constant outages because of misconfigurations and someone having to be on-call at all time.
Needing on-call at all (except for maybe security alerts, but those are usually not part of DevOps) is the result of this Google-influenced "crashes just happen" nonsense where "manager" culture is preferred over results (aka uptime).
> Real ones are oncall for the systems they build.
Code of Hammurabi's Law 229: "If a carpenter has erected a poorly constructed house, so that the walls cave in and kill the homeowner, that carpenter shall be executed."
> Real ones are oncall for the systems they build.
That’s what DevOps meant where I worked.
I remember the separation of developers from deployment being one of the things DevOps pushers wanted to reduce.
In fact it was the beginning of my career when the older senior people were already trying to get away from the assembly line style of software where each person worked on their part and then tossed it over to someone else to become their problem.
there's eventually a point in organization size where it makes sense to have some kind of centralization to make it easier for teams to actually be on call for the systems they build without grinding them to dust.
SRE at google works closely with devs to architect and build systems together. Most of them are on the SWE ladder, are able to bug fix and scale directly. They're not just ops.
The point of SRE is to have a team dedicated to scaling/resilience to balance the tension between product pushing to ship features.
If SRE cannot throttle the release cadence or fix bugs or change architecture in binaries, it's not really SRE.
> SRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
You're conflating a whole lot of different models here. In fact, "developers should be on call for the systems they build" is a principle that most schools of DevOps thought emphatically subscribe to.
I call that job SysOps - caring about systems and operation
SRE just came from the big guys calling their sysadmins SRE
Sre seems to be disappearing anyway in favor of infrastructure and platform engineering. Which seems to just be kubernetes admin jobs using terraform.
there is always infra/platform underneath the system, they are the SREs
That doesn't apply when your org is big.
DevOps was a reasonable idea, but it rarely actually manifested. If a company had an actual "DevOps group" separate from the developers, by definition they weren't doing DevOps.