I would recommend reading "Africa is Not a Country" that delves into this exact question. It is the pinnacle of self-serving arguments for European countries to first spend centuries colonising Africa, stolen its wealth and cultural artifacts, enriching themselves in the bargain, and dividing the continent in a way that would guarantee political instability, to then use that same instability as an excuse for not returning those cultural artifacts.
Of course this article is about Indonesia - that's not a country that screams instability in any case
To be honest, artifacts are not wealth. Resources are wealth and you do not use artifacts as resources. They can be a display of wealth but not the wealth itself.
Having been multiple times at the Chirac Quai Branly museums, it's pretty clear the argument is valid: most pieces seem totally forgettable and would likely have been discarded by their owners. Heck, even the "royal door of XXX king palace" is a 1-meter wooden door with carvings that aren't particularly fine. Those artifacts still exist and have value because they are exposed in a museum in Paris, otherwise nobody would care.