- the artifact is likely to be seen by significantly more people and serves a much greater purpose to expose/educate those people (from other cultures) to/about the culture from which it came
Also, in a lot of cases people living there have no relation to people of culture that the artefact has originated from. And usually the way that happened was not much better than what British did with their colonisation of these countries later on.
That seems more like a judgement call that should be made by the legitimate owners of the artifacts (I liked in the other comment, that there was a focus on the problem of figuring out who the legitimate owners were, and the practicalities of getting them the artifacts).
That argument wears thin very quickly, especially when the people of the culture from which the artifact originates are not able to view it (because it now lives in London instead of their home country), and thereby learn about their own history. See for example the Benin Bronzes; imagine that the original US Constitution document were housed in a museum in Nigeria.