That argument wears thin very quickly, especially when the people of the culture from which the artifact originates are not able to view it (because it now lives in London instead of their home country), and thereby learn about their own history. See for example the Benin Bronzes; imagine that the original US Constitution document were housed in a museum in Nigeria.
The Benin Bronzes if returned will go to the descendants of the original owners - the kings of a kingdom built on slavery.
I certainly know many people in countries from which these things were taken who think they are safer somewhere stable - I have heard exactly the comment that returning things will probably mean then end up stolen by politicians from Sri Lankans with regard to the things the Netherlands returned to Sri Lanka.
Also, consider what would have happened if the things from what is now Iraq had been there at the mercy of the likes of ISIS.
In many cases the people know occupying a territory have a different culture and history to the ancient people who made something. They may even have been the conquerors who destroyed the culture that made artifacts.