alt Hacker NewsAt one of the top tier 1 ISPs in the world, there was a bastion host that allowed 2 teams of network engineers unfettered access to everything; once your permissions allowed you access to the bastion, you had everything. 50 some people with trivial credentialed access to network infrastructure that the world ran on; fatfinger a bgp config and you could take down countries. Swathes of cities were regular casualities of config mistakes, and if you locked yourself out without setting a reload in 5, it'd take an hour to get someone deployed.
That experience shattered my idea that the world was being operated by competent engineers and technicians, governed by sane policies, under the watchful care of good, knowledgable people.
The world is held together by beliefs and expectations and bubblegum and duct tape, and a few thousand people madly scrambling to keep it all running.
Replies
> That experience shattered my idea that the world was being operated by competent engineers and technicians, governed by sane policies, under the watchful care of good, knowledgable people.
Reminds me the amount of debt that exists only as an entry in an excel spreadsheets somewhere. No database with high availability and regular backups and audit logs and access control and all of that, just a spreadsheet.
> The world is held together by beliefs and expectations and bubblegum and duct tape, and a few thousand people madly scrambling to keep it all running.
Sounds like the AWS experience
Sounds like the 90’s early ISP experience scaled up. No firewalls, everything on public IPs, text files with global credentials in clear text…