This is the kind of thing that makes me wonder how many people using these AI tools are thinking about the long game.

First it's "the model will say it can't do that". Now it's "the model will just misdirect you without telling you it's doing so". For now that's only for stuff that it thinks is developing a competing model (even if you trust it to accurately determine that), but who knows? It could be anything. Maybe it'll start silently nudging you away from certain sources of information. Maybe it'll give you inaccurate troubleshooting advice to induce you to pay for some kind of support contract from a corporate partner. Maybe it'll just subtly give out bad business advice to keep everyone else from succeeding in any way. It could be doing all that right now, for all we know. These models are a complete black box and there is no limit to the misinformation, disinformation, and malicious behavior that they could be engaging in already, let alone in the future.