logoalt Hacker News

llama052yesterday at 11:07 PM2 repliesview on HN

This is why you have circuit breakers upstream. Not on every individual instance.


Replies

unscaledtoday at 2:42 AM

In highly distributed microservice architecture, there's almost never a single upstream. In some cases you may have a couple of customer-facing entry-points (a global API gateway, and a couple of BFFs), but these are not the only paths that need to be protected.

There are client-side retries (which have broken GitHub in this case) and server-side initiated API calls between microservices that don't pass through any of your ingresses (e.g. triggered by an ETL pipeline, or a scheduled job).

With a complex architecture you can't just slap a circuit breaker on a couple of ingresses and call it a day. Don't get me wrong, putting them there does go a long way, but you won't be covering all your bases.

show 1 reply
jdm2212yesterday at 11:15 PM

Doesn't do you any good if the outage is in the circuit breaking layer, which it was for GitHub (this started as a load balancer outage).

show 1 reply