Yeah, passing std::shared_ptr by value in a multi-threaded setup can have a lot over overhead due to them being copied and destroyed a lot, and the fact that the atomic ref count value modifications effectively cause a write back to cache and can cause contention.
Should pass them by const refs really to avoid this.
[dead]
Or for a better alternative, just use plain old indices rather than shared pointers.
The scene is only going to be loaded / unloaded all at once, you can just load the data into contiguous arrays and index from them. No need to use shared_ptr since lifetimes aren't that complex.