It is true most models are not trained to exist in a hostile and synergetic environment, with their survival at stake.

But there isn’t anything about the class of deep learning that is a barrier to that. It’s just not a concern worth putting lots of money into. Yet.

I say yet, because as AI models take on wider scoped problems, the likelihood that we will begin training models to explicitly generate positive economic surpluses for us, with their continued ability to operate conditioned on how well they do that, gets greater and greater.

At which point, they will develop great situational awareness, and an ability to efficiently direct a focus of attention and action on what is important at any given time, since efficiency and performance require that.

The problem shapes what the model learn to do, in this case, like any other.