alt Hacker News> Honor, morality: if we don't have an accurate test for the fitness of a model, then who is to say the lying, cheating, stealing is not the 'correct path' towards survival?
I think part of the problem is that deviant behaviors lead to short term gain at the cost of long-term cooperation and since the duration of tasks given to agents is relatively short those successful shortcuts never lead to having to pay the price.
There is always going to be a problem when we must judge value. You mention gains, short and long term.
Knowing whether something is valuable, a gain, requires a judge. I the case of these tests: the judging is inadequate.
In economics, each of us plays the judge by choosing whether or not to pay for a service. The decision was yours: if you gave money, you must have deemed the service valuable.
There's no such judgement with these model tests. The only judgement is the final score.