alt Hacker News> I still don't totally get why the shift happened when it did. Five years ago all three camps were doing fine. Now the VM+systemd crowd has basically disappeared from job postings, serverless stayed niche, and K8s just won. > > My best guesses: managed K8s (EKS, GKE, AKS) got mature and the talent pool flipped: enough people learned it that hiring for anything else became the harder choice. And Helm made "just use someone else's chart" a real option. But I'm not certain. If you were there for the shift and have a better theory, I'd genuinely like to know.
Pretty much, almost. Have spent a bunch of time in my career working on the "VM + systemd" setups, stuff running on a rack, or in an ec2 on cloud - managed kubernetes is a lot better for me than those cobbled together messes. There's "easier" setups but usually end up costing me a lot more in time and $.
To answer simply, it became good + convenient. I could complain about plenty, and people here like to, but honestly you couldn't pay me to go back to the old way. The one legitimate gripe is the upgrade schedule is exhausting, on AWS it's about every 6 months before you go into extended support. I also hate being at the mercy of arbitrary decisions like "ok we know a huge chunk of the web going back a decade has architected off our Ingress API, but recently we decided we dont really like that way anymore and we want you to use Gateway API instead, so, um, like ya we know it just killed off one of the most used open source ingress configs (ingress-nginx) but yea trust us bro this is going to be so much better" kind of thing.
Replies
I somewhat agree with you... but it's not like you don't need some actual experts who know what they're doing, especially when stuff goes bonkers and it will go bonkers.
Even on AWS EKS, you will run into bullshit with their network overlay. Egress policies are a mess (at least half a year ago, you were not able to say something like "allow pod A to egress traffic to service (!) B" despite a service resolving down to an IP address in the end.
And that's before going into the unholy mess that is getting connectivity to and from the external world to your cluster. Cloudfront, ACM certificates, ALB, ALB-EKS integration, Route53, Route53-EKS integration, EFS, EFS-EKS integration, EBS, EBS-EKS integration, RDS, RDS-EKS integration, IAM-EKS integration, SSM, SSM-EKS integration, autoscaling... and if you want more pain and don't already wince, try setting that up across regions or, as I had to do once, across account boundaries.
Kubernetes is powerful. But do not make the mistake of assuming it's easy to get started with, at least on the admin side. Even if you got prior AWS experience, getting it all integrated into EKS so you don't have to deal with Terraform and helm/k8s for a full deployment of a piece of software will take you an awful lot of time.
For users though? It's a breeze, I will admit as much. Everything down to the firewall rules can be encoded in k8s spec files.
The upgrade cycle is a feature, not a bug. If (when) you need to do a big lift and shift, or there's some 0 day CVE, push buttan, get security update. You CAN drift behind but there's a real $$$ cost to that now. Every three months I toss opus at my k8s stack and verify it's compliant with k8s v1.xx.y and then push the upgrade button on my staging cluster, and then a week later I push the upgrade button on my prod cluster. What used to be two days of maintenance every quarter is now more like 2-5 minutes spread across the two upgrades.
I'll admit I'm dreading switching over to the gateway api, but by the time I get forced off ingresses it should be a stable/mature ecosystem. That's still a ways out though.
I don't know anyone still dealing with VMs anymore, except our IT guy who manages a couple of pet servers for random executives from the before times. In the last year k8s has started absorbing executive pet processes and the number of VMs our IT guy manages has dropped by about half.
While I'm here spouting stuff, yeah hiring for k8s is real easy, if our SRE gets hit by a bus, he can be replaced in a week, and we can probably struggle through using opus until that happens. K8s being he lingua franca of git ops IaC makes it real easy for the new guy to parachute in and start working. Every VM thing is going to be totally bespoke and have the personality of the guy who designed it, which is rarely a good thing.