Show HN: Retry a command with exponential backoff and jitter (+ Starlark exprs)

Hey, that’s funny, I wrote a blog post about the many ways you can implement such a program, and it was discussed on HN: https://news.ycombinator.com/item?id=42103200

I have a collection of small sysadming/scripting utilities distributed as a single binary here:

https://github.com/skx/sysbox

One of those is "splay" to sleep a random amount of time, before running a command. Very useful to avoid lots of things running across a fleet at the same time.

For Python, consider Tenacity: https://tenacity.readthedocs.io/en/latest/

At the CLI, this is nice for not depending on Node.

My view is that you basically never want exponential backoff.

The only time exponential backoff is useful is if the failure is due to a rate limit and you specifically need a mechanism to reduce the rate at which you are attempting to use it.

In the common case that the thing you're trying to talk is just down, exponential backoff with base N (e.g. wait 2x longer each time) increases your expected downtime by a factor of N (e.g. 2), because by the time your dependency is working again, you may be waiting up to the same amount of time again before you even retry it! Meanwhile, your service is down and your customers can't use it and your program is doing nothing but sleeping for another 30 minutes before it even checks to see if it can work.

And for what? What is the downside to you if your program retries much more frequently?

I much prefer setting a fixed time period to wait between retries (would you call that linear backoff? no backoff?), so for example if the thing fails you just sleep 1 second and try again, forever. And then your service is working again within 1 second of your dependency coming back up.

If you really must use exponential backoff then pick a quite-low upper bound on how long you'll wait between retries. It is extremely frustrating to find out that something wasn't working just because it was sleeping for a long time because the previous handful of attempts failed.

Looks similar to https://github.com/rye/eb

If I had a nickel for every time I've written exponential backoff with jitter I'd have like several nickels.

This looks cool - will give it a try (hah!) Curious on why you picked starlark instead of cel for the conditional scripting part?

Very cool project! Just a suggestion: since you do have pre-built releases on GitHub, you should mention that in the Installation section of your readme.

Typically the kind of library that is useless and root cause of the dependency hell we are now living in.

That kind of simple things should be a basic inside once program or at worse a simple snipper copied from stack overflow or anything like that

Systemd does exponential retry but IDK about jitter?

[dead]