It's plausible, but also, AI systems have a long history of being plugged in on the wrong side of security boundaries.

Remember when you were able to take over any Instagram account by asking the helpful support AI to send a password reset to an email address of your choice?