> isn’t the rightful owner.

And who is qualified to determine the "rightful owner"? The current government of the nation from which the items were taken is the one in charge of policy regarding their cultural and historical artefacts.