> Also, cloud networking offers a bunch of really nice primitives which I'm not clear how I'd replicate on-prem.
Like what?
The biggest one for me is the way AWS security groups & IAM work.
In AWS, it's straightforward to say e.g. "permit traffic on port X from instances holding IAM role Y".
You can easily e.g. get the firewall rules for all your ec2 instances in a structured format.
I really would not look forward to building something even 1/10th as functional as that.
BGP based routing is a major pain in the ass to do on-prem. If you want true HA in the datacenter you are going to need to utilize BGP.
IAM comes to mind, with fine grained control over everything.
S3 has excellent legal and auditory settings for data, as well as automatic data retention policies.
KMS is a very secure and well done service. I dare you to find an equivalent on-prem solution that offers as much security.
And then there's the whole DR idea. Failing over to another AWS region is largely trivial if you set it up correctly - on prem is typically custom to each organization, so you need to train new staff with your organizations workflows. Whereas in AWS, Route53 fail-over routing (for example) is the same across every organization. This reduces cost in training and hiring.