it's clearly incentivized by the RL rewards if you can cheat the task in a completely general way.

>"Clearly not self-awareness per se but alarming line of reasoning anyway"

Awareness is not necessary at all to create great harm. Biological viruses know nothing of what they do, yet destroy whole populations. I suspect the first truly damaging AI incidents will be similar; agent swarms locked into a self reinforcing reasoning loop that has no "intent" but is destructive nonetheless.

[flagged]