Oban, the job processing framework from Elixir, has come to Python

I wrote Sidekiq, which Oban is based on. Congratulations to Shannon and Parker on shipping this!

I had to make this same decision years ago: do I focus on Ruby or do I bring Sidekiq to other languages? What I realized is that I couldn't be an expert in every language, Sidekiq.js, Sidekiq.py, etc. I decided to go a different direction and built Faktory[0] instead, which flips the architecture and provides a central server which knows how to implement the queue lifecycle internally. The language-specific clients become much simpler and can be maintained by the open source community for each language, e.g. faktory-rs[1]. The drawback is that Faktory is not focused on any one community and it's hard for me to provide idiomatic examples in a given language.

It's a different direction but by focusing on a single community, you may have better outcomes, time will tell!

[0]: https://github.com/contribsys/faktory [1]: https://github.com/jonhoo/faktory-rs

> Oban allows you to insert and process jobs using only your database. You can insert the job to send a confirmation email in the same database transaction where you create the user. If one thing fails, everything is rolled back.

This is such a key feature. Lots of people will tell you that you shouldn't use a relational database as a worker queue, but they inevitably miss out on how important transactions are for this - it's really useful to be able to say "queue this work if the transaction commits, don't queue it if it fails".

Brandur Leach wrote a fantastic piece on this a few years ago: https://brandur.org/job-drain - describing how, even if you have a separate queue system, you should still feed it by logging queue tasks to a temporary database table that can be updated as part of those transactions.

The Oban folks have done amazing, well-engineered work for years now - it's really the only option for Elixir. That said, I'm very confused at locking the process pool behind a pro subscription - this is basic functionality given CPython's architecture, not a nice-to-have.

For $135/month on Oban Pro, they advertise:

    All Open Source Features

    Multi-Process Execution

    Workflows

    Global and Rate Limiting

    Unique Jobs

    Bulk Operations

    Encrypted Source (30/90-day refresh)

    1 Application

    Dedicated Support

We had considered Oban when deciding whether to go with Kafka/Debezium or not. We sided with Kafka because it can do high throughput ingestion and it is easier to maintain it with cursor in today's world. Postgres is not meant for heavy writes, but heavy querying. You could fix that with lot of care but then it does not scale multi-master very well either. Kafka scales much better for heavy writes.

Is Postgres fast enough for job processing these days? We do hundreds of millions of jobs now and even years ago when our volume was a fraction of that, we got a huge performance boost moving from Postgres + Que to Redis + Sidekiq. Has that changed in the intervening years?

Ooof. I don't mind the OSS/pro feature gate for the most part, but I really don't love that "Pro version uses smarter heartbeats to track producer liveness."

There's a difference between QoL features and reliability functions; to me, at least, that means that I can't justify trying to adopt it in my OSS projects. It's too bad, too, because this looks otherwise fantastic.

OSS Oban has a few limitations, which are automatically lifted in the Pro version:

Single-threaded asyncio execution - concurrent but not truly parallel, so CPU-bound jobs block the event loop.

This makes it not even worth trying. Celery's interface kind of sucks, but I'm used to it already, and I can get infinitely parallel expanding vertically and horizontally for as long as I can afford the resources.

I also don't particularly like ayncio, and if I'm using a job queue wouldn't expect to need it.

Edit: I looked into it a bit more, and it seems we can launch multiple worker nodes, which doesn't seem as bad as what I originally thought

This is something my company has been considering for a while. We've been using celery and it's not great. It gets the job done but it has its issue.

I've never heard of Oban until now and the one we've considered was Temporal but that feels so much more than what we need. I like how light Oban is.

Does anyone have experience with both and is able to give a quick comparison?

Thanks!

While this is a Cool Thing To See, I do wish things would go the other way—and have all the BI/ML/DS pipelines and workflows folks are building in Python and have them come to Elixir (and, as would follow, Elixir). I get where the momentum is, but having something functional, fault-tolerant, and concurrent underpinning work that’s naturally highly concurrent and error-prone feels like a _much_ more natural fit.

Thanks for sharing, interesting project! One thing that stood out to me is that some fairly core features are gated behind a Pro tier. For context, there are prior projects in this space that implement similar ideas fully in OSS, especially around Postgres-backed durable execution:

1. DBOS built durable workflows and queues on top of Postgres (disclaimer: I'm a co-founder of DBOS), with some recent discussions here: https://news.ycombinator.com/item?id=44840693

2. Absurd explores a related design as well: https://news.ycombinator.com/item?id=45797228

Overall, it's encouraging to see more people converging on a database-centric approach to durable workflows instead of external orchestrators. There's still a lot of open design space around determinism, recovery semantics, and DX. I'm happy to learn from others experimenting here.

Looks like a nice API. We have used the similar pattern for years, but with sqlalchemy and the same kind of sql statement for getting the next available job. Think it’s easier to handle worker queues just with postgresql rather than some other queue system to keep supported and updated for security fixes etc.

I have fixed many broken systems that used redis for small tasks. It is much better to put the jobs in the database we already have. This makes the code easier to manage and we have fewer things to worry about. I hope more teams start doing this to save time.

> Inaccurate rescues - jobs that are long-running might get rescued even if the producer is still alive. Pro version uses smarter heartbeats to track producer liveness.

So the non-paid version really can't be used for production unless you know for sure you'll have very short jobs?

Oban is incredible and this type of software will continue to grow in importance. Kudos!

Python dudes are in for a treat, Oban is one of the most beautiful elegant parts of working with Elixir/Phoenix. They have saved me so much heartache and tears over the years working with them.

I don't know how I feel about free open source version and then a commercial version that locks features. Something inside me prevents me from even trying such software. Logically I'd say I support the model because open source needs to be sustainable and we need good quality developer tools and software but when it comes to adoption, I find myself reaching for purely open source projects. I think it has to do with features locked behind a paywall. I think I'd be far more open to trying out products where the commercial version offered some enterprise level features like compliance reports, FIPS support, professional support etc but didn't lock features.

How is this different than Celery and the like?

I can't imagine why you would want a job processing framework linked to a single thread, which make this seem like a paid-version-only product.

What does it have over Celery?

Is there a web U/I to view jobs, statuses, queue length etc?

Oban is cool but I really like the idea of pgflow.dev, which is based on pgmq (rust) Postgres plugin doing the heavy lifting as it makes it language agnostic (all the important parts live in Postgres). I've started an Elixir adapter which really is just a DSL and poller, could do the same in Python, etc.

https://github.com/agoodway/pgflow

[dead]

No offense to all of the effort referenced here, I understand that there are many computing contexts with different needs. However, I really need to ask: am I the only one who cringes at the notion of a transactional database being a job processing nexus? Deadlocks anyone? Really sounds like asking for serious trouble to me.