but consider for a moment a world in which they have access to internal, non-lobotomized models, have run evals on them and have been genuinely concerned by what they saw.

That world arrived with GPT-2: https://techcrunch.com/2019/02/17/openai-text-generator-dang... , https://www.weforum.org/stories/2019/02/amazing-new-ai-churn...

These people have been distinctly unreliable at predicting the robot apocalypse, yet they demand a degree of control that shouldn't be entrusted to a genuine psychic oracle.