alt Hacker News> It is not designed to give you the most accurate answer, it is designed to give you the most likely series of words following your prompt.
That's not specifically true either; training is more complex than that. ChatGPT had to be trained, for example, to answer questions in a chat format.
> If an LLM is trained on 10 jackasses thinking bleach is a medicinal drink..
Again with the hypotheticals! You literally cannot discuss this subject without hallucinating things that don't exist. LLMs are trained on huge corpus of information from books to videos to reddit posts. Ultimately, statistically, it's going to predict the most common answer to something. Yes, that might be wrong but the vast majority of the time it's going to be right. And you know what, in the real non-hypothetical world, it works great. As much you don't want it to. You can hypothetically hallucinate as many weird unlikely scenarios as you want but that doesn't make it true.
I see there is no point trying to get through to you. These are not hypotheticals, as an example just recently ChatGPT advised a user to mix bleach and vinegar. I have seen Googles AI hallucinate many details that I had to double check and would've been disastrous had I assumed it was truthy. There's a reason why models have the legal clauses like 'this is for entertainment purposes only' because they are trying to shield themselves from legal obligations when it makes catastrophic mistakes.
The people least willing to understand how the system works are also the most willing to blindly believe it.