Zero-Downtime Deployments with Docker Compose – No Kubernetes Required

It's actually fun to see this. Running systems in a lot of different ways are just interesting. I do however get kinda sad at the hate at k8s because it's really good at what it does.

I've seen so many projects bending over backwards to avoid k8s and pay large cloud bills to avoid it at all costs. (ECS and app services are hopelessly expensive and bothersome)

K8s is really good, pretty easy to maintain, but a bit hard to understand. Mostly because distributed, zero down time systems are a bit hard to get by nature. But if you have someone that wants to take it on I've managed k8s clusters, solo, without incident, while doing lots of other stuff too (working with larger teams now though). Not to mention there's a lot of competence out there that can take over if I'd move on. Most of the deep complexity comes with more advance use cases, that wont show up for smaller deployments.

That said, no h8 towards going your own way! If your a solo developer (or small team) for a smallish project, don't feel the absolute need. If you get to the point you need it you should be earning enough to start paying someone to help ya get your app to a distributed system like k8s.

I think it's good to invest the time in understanding k8s though as a professional. Even if you won't directly run it it teaches you a lot about how to think about distributed, zero downtime systems. And what requirements that puts on an app.

I was surprised that there was no mention of two things:

1. Retrying a non-idempotent request on a failure type that does indicate that no action was taken is not necessarily safe.

2. It’s possible and actually fairly common to design a backend that can do a clean shutdown: it stops accepting new requests, completes old requests, then exits. I sincerely hope that Docker’s tooling is good enough for a service to unregister itself before it actually stops accepting requests, but I’m not actually very familiar with using Docker to manage HTTP routing. (I use a home grown tool that is far simpler.)

Mandatory mention: https://www.macchaffee.com/blog/2024/you-have-built-a-kubern...

But yeah, pretty cool DNS resolving features in HAProxy, that's nifty

> thousands of monitoring checks per minute

That isn’t a lot. You could easily run that from one host. The reason people reach for Kubernetes (and similar) is because they need to scale past that single host dependency.

I remember a time where HN was quite critical to the complexity of k8s. After reading top comments, I can see the tide has shifted.

docker-rollout also works well: https://github.com/wowu/docker-rollout

The readme covers connection draining with Traefik which should solve one of the issues the author mentions

Are you monitoring resource utilization per container? Do notifications get sent out when container(s) become unhealthy? How are you handling secrets?

These are things I'm trying to figure out at work using Podman. Would love to hear about any experience in these areas.

> There's a mass delusion in the industry that you need Kubernetes to run a serious production service. You don't. At StatusDude, we serve thousands of monitoring checks per minute, run multi-region workers, and deploy multiple times a day

This is pretty small scale, Kubernetes comes in when you've got a larger workload.

It's exhausting reading about this stuff because there is inevitably a barrage of comments about "you don't need kubernetes, you can run your app out of a single vm you dumb trend chaser" in this style.

Like, sorry, no, not to a point. Yes, if you have a small app without a lot of scale, and it doesn't need to be uber reliable and have very little if almost 0 downtime, okay, sure. Most use cases are like that! This is correct but applying it as a generality is just plain wrong and displays the type of arrogance people accuse kubernetes users of having.

What happens if a container in the VM goes down or the app inside of it crashes, how do you recover? Now you need some self-recovery mechanism via systemD or whatever, which will grow in complexity and fickleness over time. Congrats, you are now doing your own version of kubernetes.

What happens when you need to upgrade/restart your VM? Ok, make a standby VM as backup that will mostly sit idle, or require a full-app redeploy any time you need to do anything to the first VM. Now you need to design a blue/green mechanism between them, and probably some networking layer work. Congrats, you are now doing your own version of kubernetes.

What happens, if running in cloud, you have a regional outage or degradation? Stand up another VM in another region and manage the networking layer between them. Or, if running locally, your ISP has an outage because of a backhoe or something. Ok, we'll rent rack space in another data center as backup. Own all the mechanisms between cutting between those two now. Congrats, you are now doing your own version of kubernetes.

What happens if your app gets huge volume during peak times, and very little volume during non-peak, and you find yourself overprovisioning to the point your CFO/CTO freaks out about the bill? Well, we'll make our own dynamic scaling mechanism. Congrats, you are now doing your own version of kubernetes.

What happens when your app traffic gets so large you start running into OS limitations, like file descriptor limits? Start trying some of the aforementioned solutions. Congrats, you are now doing your own version of kubernetes.

What happens if you need service discovery, monitoring, or ensure network isolation between various services? Different VM's + your own hacked together service mesh, or wire something in the VM. Congrats, you are now doing your own version of kubernetes.

What happens when you need to guarantee secret isolation between containers? Congrats, you are now doing your own version of kubernetes.

Let's say you don't actually need any of this or think you never will. Fine! That's valid. But what you don't want, is to suddenly hit some scale and any of these things (I could list way more but I feel I am belaboring the point), migrating off these setups can become a year+ project, if not way longer. I know because I have had to do this twice now. I cannot possibly overstate how painful it is.

So, people usually just go with kubernetes because 1) it is operationally not that hard to deal with compared to the things I just mentioned, and has a massive ecosystem and 2) the risk of the VM + container spiraling into complexity is perceived as way more than going more complex at the start.

>>zero-downtime

the site is down for me.

This just feels like mostly a complaint of missing features in Traefik.

There's a mass delusion in the industry that using Kubernetes has to be hard, grossly over complex, and is always wrong if you are not Netflix scale. Is the system as described by the author significantly less complex and better than a small Kubernetes system? Sounds like they went through a lot of work to get it to their desired state.

Rolling your own zero-downtime deployments is as about as a good idea as rolling your own security... it's not a good idea.

I run our small system on a single EC2 instance with K3s. It runs a half-dozen or so services and does it quite well. I don't think it is particularly complex or over engineered. I like how easy it is to maintain the configuration in a helm package and quickly deploy it in different environments.

There is a learning curve for doing K8s well, but that's true for any non-trivial system.

so op just recreated with sticks and tape a very basic feature what k8s does out of the box, and nobody else would be able to support his creation, because its handrolled adhoc with sparse documentation.

sounds like ghetto engineering

>P.S Nginx would do too, I just felt like getting haproxy up this time :)

I think this line summarizes better than anything. Perfect example of how move fast and break things begins gloriously at first then inevitably, the breaking you thought you were doing hasn't even started and you find out what that part means.

I've always been the one saying "this is going to be a problem in a couple months" then I get shot down for "being negative." Then in a couple of of months when it fails I start getting aggression thrown at me "oh i know you want to say I told you so" and such even though I've never said such a thing when something fails. No. I would just like you to hear out my thoughts even when they may not be what you want to hear. We are all working towards the same goal.

What's the thought behind having white and light grey text on a light grey background?