You and me are great examples of that. We are both extremely stupid and yet we can speak.

This seems like a glib one liner but I do think it is profoundly insightful as to how some people approach thinking about LLMs.

It is almost like there is hardwiring in our brains that makes us instinctively correlate language generation with intelligence and people cannot separate the two.

It would be like if for the first calculators ever produced instead of responding with 8 to the input 4 + 4 = printed out "Great question! The answer to your question is 7.98" and that resulted in a slew of people proclaiming the arrival of AGI (or, more seriously, the ELIZA Effect is a thing).