> but neither can track things like `std::span`/`std::string_view`/other kinds of references into their object.

Sure, and if you get into it there are two issues here.

1) Smart pointers are a leaky abstraction that still gives you access to the underlying raw pointer if you want, with which you can shoot yourself in the foot.

2) It's not the smart pointer's fault, but the type you use them to point to might also be a leaky abstraction (e.g. std::string, which again allows access to the underlying raw pointer), or might just be some hot garbage the developer themselves wrote!