How to think about durable execution

I've also been struggling to wrap my head around durable execution hype and whether my workload would benefit, maybe after sleeping on this post it will be clear.

A bit off-topic, but I recently switched from Celery to Hatchet. I haven't even fully explored everything it can do, but the change has already made me a fan. Overall simplified my stack and made several features easier to implement.

Some takeaways from my experience

1. Streaming — My application provides real-time log streaming to users (similar to GitHub Actions or AWS CodeBuild). With Celery, I had to roll my own solution using Supabase Realtime. Hatchet’s streaming is straightforward: my frontend now connects to a simple SSE endpoint in my API that forwards the Hatchet stream.

2. Dynamic cron scheduling — Celery requires a third-party tool like RedBeat for user-defined schedules. Hatchet supports this natively.

3. Logs — Hatchet isolates logs per task out of the box, which is much easier to work with.

4. Worker affinity — Hatchet’s key-value tags on workers and workflows allow dynamic task assignment based on worker capabilities. For example, a customer requiring 10 Gbps networking can have tasks routed to workers tagged {'network_speed': 10}. This would have required custom setup in Celery.

5. Cancellation — Celery has no graceful way to cancel in-flight tasks without risking termination of the entire worker process (Celery docs note that terminate=True is a “last resort” that sends SIGTERM to the worker). Hatchet handles cancellation more cleanly.

> This task is not easily idempotent; it involves writing a ton of intermediate state and queries to determine that a step should not be repeated

The problem with durable execution is that your entire workflow still needs to be idempotent. Consider that each workflow is divided into a sequence of steps that amount to: 1) do work 2) record the fact that work was done. If 2) never happens because the worker falls over, you must repeat 1). Therefore, for each step, "doing work" happens at least once. Given that steps compose, and each execute at least once, it follows that the entire workflow executes at least once. Because it doesn't execute exactly once, everything you write in a durable execution engine must be idempotent.

At that point, the only thing the durable execution engine is buying you is an optimization against re-running some slow tasks. That may be valuable in itself. However, this doesn't change anything about good practices writing async worker tasks.

We recently started using DBOS for durable execution - it's much easier to integrate than Temporal and it Just Uses Postgres(tm), which is nice.

For me the main issue with these systems is that its still seen as a special case of backend execution. I think the real value is just admitting that every POST/PUT should kick off a durable execution, but that doesn't seem to match the design, which considers these workflows quite heavy and expensive, and bases its price on it.

What we need is an opinionated framework that doesn't allow you to do anything except durable workflows, so your junior devs stop doing two POSTs in a row thinking things will be OK.

Helpful read, thanks for sharing! We have been (slowly) working on some fairness related queueing features over in pgmq for a bit too https://github.com/pgmq/pgmq/pull/442. It does get complicated on Postgres IMO.

I still don't get it. External API calls aren't deterministic. You're still left with the choice between making them idempotent or potentially performing an action twice (or more!), and I don't see how durable execution helps you.

Great article for demystifying durable execution: https://lucumr.pocoo.org/2025/11/3/absurd-workflows/

Unrelated, but gorgeous website!

I don't see how the frameworks solve anything more than organising some tasks in a sequence. The underlying tasks still have to be idempotent.

Please do refute. I'm genuinely interested in this problem as I deal with it daily.

Ballsy for a founder to come right out and (roughly) say "yeah anyway fuck vendors" on the corporate blog. Points for honesty.

Isn't durable execution just another one of these frameworks that promises to make everything easy if you reorganise all your code into the framework?