Have experienced it, but didn’t document.

Downstream database of our edge serverless platform went down. A tonne of requests failed all at once. Every service in the microservice request path, and the client, had their own retry policy.

Clients all retried at the same time. Retries amplified in our microservice graph (1 request at the front door ended up with like 10s of retries internally as each downstream microservice along the path retried requests). Request queues backed up and couldn’t drain fast enough. Clients all timed out at roughly the same time. All waited the same time. All retried again at the same time.

It was a pulsing thundering herd of many hundreds of thousands of requests at the front door that was amplified by internal retries.

Had to tune up load shedding to 100% after the database outage was mitigated until the backend recovered then tune it down in increments to restore service.

Added jitter to clients and turned off retries on the serverless platform.