LLMs have shown us how easily humans are fooled. Just provide a similar answer to which was seen in similar circumstances, and we start misleading ourselves into it's method and "understanding".

Though perhaps I'm wrong that people use a model of the world to think through answers to queries. Maybe many people do just repeat something like whatever they heard before. In which case, LLMs make as much sense as the average ignorant human.