> Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow.

This reminds me of some research I read about in the 1980s or 1990s on perceptions of the speed of command line commands. If command time varied over a range from nearly instantaneous to say 100 ms fairly uniformly through that range, people would perceive the system as overall being faster when the researchers added a variable delay to all the commands that made them all take 100 ms.

Humans apparently really like consistency.

> When SRE attempted restart Service X, requests queued in the job queue were all retried en masse, which led to an overload of the partially-restarted service and a subsequent failure

...and this reminds me of something else, from around 1983. I was working at a small Unix workstation maker. The guy in the office across the hall found one morning that the battery for the clock on his workstation had died, and the system time had come up after boot as the Unix epoch.

He shut down, put in a new battery, booted, and then set the clock to the current time, 13 or 14 years after the epoch.

Almost immediately his hard disk light came steadily along, and he could hear the disk furiously seeking, and the system became completely unresponsive.

It turned out AT&T cron in the early '80s wasn't smart about time changes. It had tried to all at once every cron job that should have run in the 13 or 14 years that the time just jumped.