alt Hacker News>> If AI is not working for humans' benefits
>> not working towards an optimization problem set-up by humans
I am suggesting neither of the two.
Not a great analogy but a parent may be working for an infant's benefit without the infant yet being able to understand. Taking an arbitrarily broad example, the problem to optimize for could be "help humanity advance", and other aspects could be subgoals of the same including what mathematics is useful.
I see several problems with this,
- by definition, the AI isn't human, irrespective of its computational abilities, it needs a human to tell it what being human is, and the ways "humanity can be advanced" need to be validated on this basis
- "advancing humanity" isn't a one dimensional problem/single-KPI optimization game, you might optimize certain things (e.g. expected life expectancy) at the detriment of others (e.g. freedom of movement). If you are not understanding the proposition, you are not understanding the target outcome.
- even if the target outcome is perfectly formally specified and commonly understood (which it can't), you want your AI to rationalize that the journey to get there is the most direct and efficient.
- especially so since subsequent runs of the same prompt will provide different plans
- anything less than that is begging to be conned: as a sensible person, you wouldn't give unlimited power and all your faith to a single individual trusted to "advance humanity" if they cannot be understood. On what ground would you give the machine a pass?
In all, such comments really worry me. It's like AI is triggering for some people the kinds of oppressive religious feelings whereby the individual should submit itself to the will and desires of a pretended all-powerful being. Do you really think our ancestors chose to cut off their legs in abandonment when they figured that some animals could outpace them? No, they built traps and throwing weapons to catch them and see how they taste.