alt Hacker NewsHmm, you think?
I’m currently engineering a system that uses an actor framework to describe graphs of concurrent processing. We’re going to a lot of trouble to set up a system that can inflate a description into a running pipeline, along with nesting subgraphs inside a given node.
It’s all in-process though, so my ears are perking up at your comment. Would you relax your statement for cases where flexibility is important? E.g. we don’t want to write one particular arrangement of concurrent operations, but rather want to create a meta system that lets us string together arbitrary ones. Would you agree that the actor abstraction becomes useful again for such cases?
Replies
> we don’t want to write one particular arrangement of concurrent operations, but rather want to create a meta system that lets us string together arbitrary ones. Would you agree that the actor abstraction becomes useful again for such cases?
Actors are still just too general and uncontrolled, unless you absolutely can't express the thing you want to any other way. Based on your description, have you looked at iterate-style abstractions and/or something like Haskell's Conduit? In my experience those are powerful enough to express anything you want to (including, critically, being able to write a "middle piece of a pipeline" as a reusable value), but still controlled and safe in a way that actor-based systems aren't.
Data flow graphs could arguably be called structured concurrency (granted, of nodes that resemble actors).
FWIW, this has become a perfectly cromulent pattern over the decades.
It allows highly concurrent computation limited only by the size and shape of the graph while allowing all the payloads to be implemented in simple single-threaded code.
The flow graph pattern can also be extended to become a distributed system by having certain nodes have side-effects to transfer data to other systems running in other contexts. This extension does not need any particularly advanced design changes and most importantly, they are limited to just the "entrance" and "exit" nodes that communicate between contexts.
I am curious to learn more about your system. In particular, what language or mechanism you use for the description of the graph.