More dynamic cronjobs

I keep toying with the idea of writing a cron that implements a Poisson process. Say I give it a parameter of 3600; our `pcron` would ensure the jobs occur randomly but average out to once per hour, making the timing of the next run independent of the last via the memoryless property of the exponential distribution.

The next sleep interval would be calculated probably as as t = -\lambda \ln(U) (where U is a uniform random variable). This way you ensure that the probability of the job firing in the next 10 seconds is the same whether the last job finished an hour ago or just five seconds ago. But \lambda remains the average amount of time between jobs.

It’s compelling to me because it solves thundering herd problems at the architectural level, and also because it simply seems like a lot of fun to have to code very defensively against such chaos. Switching back to a deterministic schedule after surviving such chaos probably leads to a much more robust system overall.

Learned once the hard way that it makes sense to use "flock" to prevent overlapping executions of frequently running jobs. Server started to slow down, my monitoring jobs started piling, causing server to slow down even more.

  */5 * * * * flock -n /var/lock/myjob.lock /usr/local/bin/myjob.sh

Great post. And if you want some control support for your cronjobs perl App::Cronjob[1] can provide features such has exclusive locking, so a job won't run if the previous run is still going, or provide a timeout, and some options for sending mail on success or failure

[1]https://metacpan.org/pod/App::Cronjob https://metacpan.org/dist/App-Cronjob/view/bin/cronjob

Does anyone maintain a programmatically accessible list of holidays for their company? Similar to the HOLIDAYS.txt in the article, but it would allow for things like “don’t run this the day before or during a company holiday.”

I work at a company with different holidays in certain countries, which would complicate things, and require something more structured than a list of dates. But having that accessible could be useful.

Has anyone tackled that, or come across a solution?

Also check out the 'chronic' command from moreutils. No more dev nulls.

I learned something cool about cron filtering and a nice api I didn't know existed - date.nager.at

This is great! I'm sure like a lot of programmers, I had been fulfilling the requirement for similar conditional logic by having a simple recurring cron job run other code or database queries with the conditional logic that this post demonstrates can be done directly in cron.

Cool. Had no idea you could run commands inside a CRON expression.