Isn't this about children attributing human characteristics onto the LLM, rather than the LLM itself pretending to be human?
And we do this with lots of things that are not human for children (e.g. Bluey and Peppa Pig). Do we need a "this is not a human" warning for them, too?
We probably do, considering how many seem to treat it as companionship or therapy.
Do Bluey and Peppa Pig produce convincing human-like responses to virtually any human-generated input directed towards them?
If so, then uhh yeah we should definitely regulate that shit