logoalt Hacker News

theamktoday at 3:06 AM0 repliesview on HN

I think the the second part was kinda obvious? The moment I read this:

> If you’re anything like me, you would probably have said Parallel Spawn, Prefer New, and Wait are perhaps defensible, whereas Prefer Old feels weird/backward.

it was pretty obvious I was not anything like him. My intuitive answers are pretty different.

- Parallel Spawn is useful, but it's orthogonal to the rest. Even if you have 4 workers, you'll still have to worry about hitting concurrency limit once you have enough jobs. What is it even doing in this list?

- Wait is very useful for non-scheduled tasks: if user uploaded 100 files to process, you better process them all. Sometimes you need limit, sometimes you do not (let them queue for a while until devops notices and either allocate more workers or clear them and has some harsh words with consumer).

For scheduled tasks, "Wait" seems much less useful. I can come up with a reason but they are all somewhat convoluted - perhaps you are hitting 3rd-party service, and it has a quota, so you've decided to use scheduler to avoid hitting ratelimits?

- "Prefer Old" is normally the best way. You repack takes 3-8 hours, so you set your timer to "every 1 hours, skip if running already" and you can be sure that your job finishes and the next one will start.

- "Prefer New" seems almost useless. You've already spent all this effort doing the job, why are you cancelling it and throwing away the results? If you want to add a timeout, add a timeout, preferably to specific operation. For example, if there job starts by fetching data, and this fetch can be super slow, use 'Prefer Old' and put a timeout on the fetch part. This way your job won't be interrupted if the fetch just finally succeed minutes before next scheduled interval hit.

Oh, and re "If the Prefer Old semantics are not offered, you can’t really emulate them using the two primitives of regular scheduling and limiting concurrency." - you totally can. Set concurrency to 2, and add as a first thing: "fetch the list of jobs running; if there is anyone except me, exit right away".

Really, not that tricky at all.