> owning an index means I still have a piece of the pie

You're assuming within-pie transfers.

US equities in 1990's were 40% of the world market; now they are 80%. It's pretty easy to imagine them going back to 40% or less as capital moves elsewhere. But owning indexes across currencies and markets is another layer of risk.