A skeptic's first contact with Kubernetes (2024)

I especially enjoyed where you discovered that K8s runs on string interpolation. That feeling never goes away, FYI; every time you have to play with nindent, you lose a little piece of your sanity.

I've written at length about my disdain for text interpolation in helm.

I've done some experiments using Apple's Pkl and CNCF's KCL, which look to solve these problems with a simple "configuration language".

I don't think either have very widespread adoption, though I did hear a while back that Apple is using Pkl internally.

> Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU"

Since Kubernetes 1.23, there's support for using "custom metrics" to control HPAs: https://kubernetes.io/blog/2026/07/14/custom-metrics-exporte...

Is it somewhat convoluted to setup? Absolutely! But it's possible to use relatively robust and well supported built in controllers with custom metrics.

> Given the Controller pattern, why isn't there support for "Cloud Native" architectures?

you're supposed to write your own.

> Why are the storage and networking implementations "out of tree" (CNI / CSI)?

> Given the above question, why is there explicit support for Cloud providers?

the in-tree stuff came first, and they have been extracting it in pieces for years now.

Using nix to generate k8s yaml is actually quite pleasant. You avoid all the string templating bullshit and can use nix functions and the nixos module system to build proper abstractions.

Using literally anything other than helm will make your life 100x better with k8s.

Kubernetes vs SLURM: https://slurm.schedmd.com/SC23/Slurm-and-or-vs-Kubernetes.pd...

> Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU"

I think you found the answer to that below - KEDA

K8s is a reconciliation engine for desired state, like many other adjacent tech in the infra space. Hating on yaml, json, missing types, or templating layers that drive the engine is kind of silly, and usually missing the point.. like hating dbs because you take issue with a sql layer or an orm layer. Disliking one or the other isn’t a deal breaker that invalidates the value proposition, so you may want to consider the other one before you retreat to flat text in csvs.

Kubernetes can be driven with tofu and hcl.. make it operator and even driven.. use any of many sdks, cdks to drive it, or add a completely different layer for faas or orchestration if that’s the relevant vocab for your problem space. It all runs on k8s.. that’s kinda the point.

Thanks for writing this up. The prose was really pleasant to read, a treat in this day.

YAML abstracting YAML abstracting YAML abstracting..