But my bank has no physical buildings whatsoever?

And also, how is my bank supposed to safely generate and distribute a key that we can exchange securely if an organisations who's primary purpose to generate these keys couldn't even manage to do so.

I have plenty of other services I use too. Suddenly my phone company, doctors, supermarket has to create and distribute keys for me to go around collecting so that I can securely talk to them and we have to do this in a way that stops anyone from swapping those keys out before they get to me.

I mean the current system isn't perfect, but it's a Utopia compared to that.