> there's no referee, most everyone does their best to just play hard and follow the rules

Only in small groups. Any sufficiently large groups will have a number of people (possibly sociopaths) that will break the rules for personal gain. And in a lot of cases it's enough if very few people do that.