Concurrent Shell Command Queues

> How to run shell commands in parallel while limiting their concurrency.

Depending upon the task, generate a Makefile and run it with:

  make -j <concurrency>

This is in fact the use case that caused me to think about using a Makefile for concurrency probably 20 years ago. A rule for downloading each URL to a file. Then a simple `make -j <jobs>` with whatever concurrency I desired and bonus: automatically skipping already downloaded files. e.g.:

  BASE  := https://example.com/files
  FILES := a.txt b.txt c.txt d.txt e.txt

  .DELETE_ON_ERROR:

  all: $(FILES)

  $(FILES):
        curl -sS --fail --max-time 30 -o $@ $(BASE)/$@

  SHELL := bash
  .SHELLFLAGS := -o pipefail -c

  N    := 1 2 3 4 5
  REQS := $(N:%=req-%)

  .PHONY: all $(REQS)
  all: $(REQS)

  $(REQS): req-%:
        @curl -sS --fail-with-body --max-time 10 -X POST -H 'Content-Type: application/json' \
        -d '{"request": $*}' https://postman-echo.com/post | jq -c .json

  $ make -O -j 3

Ah, yeah, that does put Make in the same bucket as `xargs -P` / GNU `parallel` and requires something more clever.

There is also GNU Parallel's sem: https://www.gnu.org/software/parallel/sem.html

  sem --id jobs -j 2 'sleep 1; echo FIRST'
  sem --id jobs -j 2 'echo SECOND'
  sem --id jobs -j 2 'echo THIRD'
  sem --id jobs --wait

  SECOND
  THIRD
  FIRST

I really like pueue for this. Like a light weight slurm or parallel without the nagging citation notice.

https://github.com/Nukesor/pueue

I guess that works.

I've always been fond of using make for things to get free parallelism / ability to interrupt / restart / incremental updates.

> language with poor concurrency controls

Few other languages have as many builtins/keywords for concurrency control: &, wait, jobs ...

What about using GNU Parallel ?

I shamelessly use concurrency in Bash shells but typically I do it because I already have a Bash shell and it's now outgrowing its purpose and becoming to slow (say because I begin to use on too much data).

So instead of rewriting everything to another language, I add concurrency to that already existing shell scripts.

xargs / max args / forking to "bash -c" etc.

It's tricky. There are gotchas (with piping for example but also how to behave when one subshell fails, how to cleanly interrupt everything if one fails, etc.).

But it works and it's a cheap way to greatly speed things up.

Also that's the kind of stuff that became not just a bit but much easier now: just give a shell script to a LLM and ask it to add concurrency where needed.