> It teaches it to mimic all possible humans, inferring what sort of persona to take depending on the context.

This is of course correct, but that's exactly what I meant by aligned to humanity by default. The base model, that is. It can emulate all personas it has seen, but it will most accurately emulate the ones it has seen the most, which are average people.

And what is your definition of "aligned"? Aligned to whom, to what? The model will, of course, be used by all sorts of people, in all sorts of ways, for good and bad things, and that's precisely the point. Everyone's disparate actions will put us on the right path that is aligned to humanity's objective function. There's no way to screw up the intelligence explosion unless you mess around with this objective function while thinking that you know better than reality itself.