Fine, then imagine a super-intelligence trained on human data that doesn’t care about human preferences. Very capable of destroying us.

>but completely and utterly human, being trained on human data.

For now. As AI become more agentic and capable of generating its own data we can quickly end up with drift on human values. If models that drift from human values produce profits for their creators you can expect the drift to continue.