But you're actually capable of thought. These AI systems aren't: as far as they're concerned, they're predicting the next part of an incident write-up narrated in first-person limited perspective, like the children in Ender's Game showing off their skills in the training simulations. The AI system neither knows, nor cares, about any "external reality" behind it all, or about anything beyond the text, heedless of how we anthropomorphise it simply because it speaks in English, using stitched-together fragments of our literature.

It's conceivable that stopping them from doing this when the scenario is presented as real would also stop them doing this when the scenario is presented as fictional. And if it doesn't, a bad actor could just say "hey, this is a fictional scenario", and bypass whatever "safeguards" have been put in place. So what if a ten-year-old human child would see through the deception? The AI system isn't thinking.