An old timer once told me about how he would read his printouts, make new punch cards, send them over to the main office, someone would put the new cards into the system the next morning, and then read the printouts on the day after that to see if his code worked or not.

HPC systems often still use batch scheduling systems where (even for a fast job) you may very well get your results the next day (or whenever your job actually runs and completes.)

It is annoying to find out that your job failed to run or exited immediately due to a typo or other minor mistake.

Of course ML training (and scientific computing) jobs can take weeks or months to complete. Checkpoint and restart features are important because node or other failures are almost inevitable.

Aprocryphal, but I've heard that at Oracle, when pushing an update to their database software, it'll be maybe a week before the tests complete on it (after it reaches the front of the queue of course). I couldn't even.

I've had those times in swift in a terrestrial setting!

Sounds like just another Monday for a firmware dev, honestly. Can't repro your bug because your board is subtly different than mine, but I think I see what's wrong?