A lot of people would have been better off if they hadn't. This argument only works if you value the lives of Europeans (and their descendants, who often were equally racist against the indigenous peoples) over the native inhabitants.