> On the other hand, when I use LLMs, I regularly see them confidently claim information confidently that is false

When talk to people, I regularly see them confidently claim information that is false as well.

This is not the point where LLMs become useless. LLMs become useless when we understand that we are social creatures and that many times the road is more important than the destination. Also, when you provide instant-travel to all the destinations, the destinations lose their value as well.