>Damned if you do, damned if you don't.
In my opinion, if the company is allowed to see the data and train on it, then they are also responsible for reporting stuff like this. Without knowing the data licensing agreement the lady had with Anthropic, if she agreed to letting Anthropic see her data, then they should do stuff like this. If she didn't agree, then I wouldn't condemn Anthropic for failing to report an attack
The second order effects of what you are suggesting are a total surveillance state.
Unworkable in practice.
You would expect them to enforce every possible legal standard, in every jurisdiction.
Why stop with criminal law? It should flag torts too, right?
Absurd.