Bridging Elixir and Python with Oban

We have a similar use case. All Elixir code base, but need to use Python for ML libraries. We decided to use IPC. Elixir will spawn a process and communicate over stdio. https://github.com/akash-akya/ex_cmd makes it a breeze to stream stdin and stdout. This also has the added benefit of keeping the Python side completely stateless and keeping all the domain logic on the Elixir side. Spawning a process might be slower compared to enqueuing a job, but in our case the job usually takes long enough to make it irrelevant.

Oban is great!

Very nice, Oban is great. I effectually found a similar approach with pgflow.dev (built around pgmq) - but the stateless deno "workers" are pretty unreliable and built an elixir worker (https://github.com/agoodway/pgflow) that can pick up and process jobs that were created by pgflow's supabase/typescript client. So maybe there's an opportunity also with Oban to have a TypeScript/Node client that can insert jobs that Elixir/Python Oban can pick up. Also, I wonder if another approach vs the python workers picking things up is to have elixir workers call/run python/lua, etc code or is that too limiting?

[dead]

I absolutely love Elixir, but if this is the bridge you need to cross, just write it in Python in the first place.

I don't see the point of Elixir now. LLMs work better with mainstream languages which make up a bigger portion of their training set.

I don't see the point of TypeScript either, I can make the LLM output JavaScript and the tokens saved not having to add types can be used to write additional tests...

The aesthetics or safety features of the languages no longer matter IMO. Succinctness, functionality and popularity of the language are now much more important factors.

I feel like if you need to utilize a tool like this, odds are pretty good you may have picked the Wrong Tool For the Job, or, perhaps even worse, the wrong architecture.

This is why it's so important to do lots of engineering before writing the first line of code on a project. It helps keep you from choosing a tool set or architecture out of preference and keeps you honest about the capabilities you need and how your system should be organized.