alt Hacker NewsWhen you have an outage, you should not retry at all. Exponential backoff is exactly how you get cascading outages. If service A fails a request to service B and decides to exponentially back off, now service A is holding open an end user request that will claim resources on service A. Fast forward ten minutes and the service B degradation has metastasized into a service A degradation. And even after service B has recovered, service A might still be dead.
To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients. Then service A knows service B is dead, does not retry, and proapgates the failure to clients. This is hard to do perfectly, but there's no alternative that works.
Replies
andrekandre • today at 12:50 AM
> To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients.
➕ show 1 reply
> To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients.
Even this is not enough, since you cannot always reliably know whether service B is dead or suffers an intermittent issue that can be safely retried just from looking at a single failure.
The classic solution, in the monolith/few-services world would be a circuit breaker. High failure rates on any service trigger a circuit breaker in the services calling it, and they'll wait for a cooldown period before trying again.
When you move to a massive microservice architecture with hundreds or thousands of microservices, setting up circuit breakers manually becomes very hard to track and do reliably. Service meshes like Istio make this slightly easier, but they still don't let you verify that all possible paths have circuit breakers and that retries are not excessive etc.