It's harder to believe something is conscious or threatening to achieve word domination once you understand that it's a machine that statistically figures out which word should come next.

The problem is, these models challenge our definition of "consciousness." Or at least they point out how hopelessly-inadequate our thinking on the subject is. Some people really, really don't like having their personal definition of consciousness challenged.

The correct response to "So what, it's just a next-token predictor" isn't a long dissertation on RLHF, training architectures, scaling laws and whatever, but rather to turn around and respond, "Sure, and how is that different from what we do?"