I think the idea is that the chances of an LLM causing death or irreparable harm to a consistently unsupervised kid are too low to be comparable with major sources of harm (like cars) and are rather on or below the level of random casual accidents, like choking or falling, where we normally start to get practical (rather than extra cautious) about the safeguards.

If true (quite probable; there aren't that many reported cases that I found-less than 40, all-time, worldwide), then we can't meaningfully distinguish between willful negligence on Anthropic's side and a "shit can happen for any reason" class situation. Especially considering those accidents seem to tend to involve various mental health issues, particularly including preexisting suicidal ideations as a comorbidity. Probably fewer than there are cases of other bona fide people talking kids into harm (verbal abuse, dares, etc.).

And if so, I guess ethics suggests that we shouldn't assume unproven malice in such a case unless there's proof of actual intent. A suspicion is not entirely without basis, but "hurting kids for profit" feels too provocative in its implications, to the extent it starts to feel misleading.