alt Hacker NewsSame experience with the interaction I described in my comment... after I finally got the correct answer, I asked where it got the faulty information from. It said it had just simply fabricated it. That's actively worse than just saying "I don't know", for something that's sold to us as an easy way to look up information.
These models are incapable of saying they don't know, because they have no concept of knowing. They simply predict the next word which is most likely.