logoalt Hacker News

sandeepkdyesterday at 10:51 PM1 replyview on HN

I am afraid thats now how infrastructure works from what I have seen. The number that really matters is QPS. For any system the QPS varies through out the day and across the week and months. Most design considerations easily absorb any 2X increase. Pick up any company and the chances are that the servers are over provisioned, no one takes chances specially with critical components.

What you have going on with Github is mix of multiple things. Traffic alone is not the cause from what little I know, it does adds to the problem for sure

1. Infrastructure is being moved to use Azure, and overall all the cloud providers are struggling with hardware at the moment (same is going on for linkedin too)

2. The core teams, the people who knew the existing systems have either been laid off or moved from Github

3. Microsoft veterans are brought in to fill the gap across the board, they are trying their best but its a lot of unknown for them


Replies

Anon1096yesterday at 11:04 PM

How much infra have you seen of the top 100 sites in the world? I've worked on multiple top 10, and absorbing a 2x increase (and the peak is very likely more than 2x) is a very very hard problem that would cause hundreds of pagers to go off and load shedding to very high degrees. There is just not tons of unused capacity lying around in wait at the scale of github. "No one takes chances with critical components" is also very wrong for the simple fact that you don't know which is the weakest link in the chain until it fails.

show 1 reply