alt Hacker NewsIMHO the main point of the article is that typical unix command pipeline pipeline IS parallelized already.
The bottleneck in the example was maxing out disk IO, which I don't think duckdb can help with.
IMHO the main point of the article is that typical unix command pipeline pipeline IS parallelized already.
The bottleneck in the example was maxing out disk IO, which I don't think duckdb can help with.
Pipes are parallelized when you have unidirectional data flow between stages. They really kind of suck for fan-out and joining though. I do love a good long pipeline of do-one-thing-well utilities, but that design still has major limits. To me, the main advantage of pipelines is not so much the parallelism, but being streams that process "lazily".
On the other hand, unix sockets combined with socat can perform some real wizardry, but I never quite got the hang of that style.