Why would I want to be constantly calling into code I have no control over, that may or may not exist, that may or may not be tampered with.

I lose control of the execution state. I have to follow the calling conventions which let my flags get clobbered.

To forego all of the above including link time optimization for the benefit of what exactly?

Imagine developing a C program where every object file produced during compilation was dynamically linked. It's obvious why that is a stupid idea - why does it become less stupid when dealing with a separate library?