alt Hacker News> Errors in those services triggered a client-side retry loop that increased traffic during recovery
Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours.
> Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
The detailed root analysis tries to pass this off as a "bug". You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour is functioning exactly as designed. In this case aggressively to try and hide problems if token service responses become flakey.
Replies
Maybe the retry logic was vibecoded instead of using an existing hardened library. After all, according to Twitter, nobody is looking at the code anymore.
Except the other side of this is interrupting a service which would otherwise have succeeded: there's a lot of unattended or minimally attended processes where an interruption is just asking the user to do the only thing they were going to do anyway - retry it.
In GitHub's case this is especially relevant - the only reason to throw an error message at the user is the hope they - the human - give up and walk away (or you break all the CI/CD builds and the time it takes humans to hit "retry" gives you some breathing room).
[flagged]
A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded).
If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go great. But there are lots of situations where one or both of those breaks down.