C++11 totally could have started skipping destructors for moved out values only. They chose not to, and part of the reason was safety.

I don't agree with you about shared_ptr (it's very common to use it for a small number of large/collective allocations), but even if what you say is true, it's still a part of C++ that focuses on safety and ignores performance.

Bottom line - C++ isn't always "unsafe-but-high-perf".