alt Hacker News> But it's not going to do that. It's literally designed to give the best and most accurate answer that it can.
No, no, no and no. This is the biggest mistake I see people consistently make with LLMs. It is not designed to give you the most accurate answer, it is designed to give you the most likely series of words following your prompt.
If an LLM is trained on 10 jackasses thinking bleach is a medicinal drink and 1 doctor who disagrees, it will by virtue of probability tell you to drink bleach. Companies add additional safeguards or system prompts to try and keep it 'on rails' but it's all probability based on what you prompt it. It is by the literal functionality of how it works to do so. A function depending on what said company ingests during training, many of which include the entire corpus of the internet.
If you do not understand how LLMs work under the hood then yes, you shouldn't trust your own personal learned experience because you've already demonstrated that you're wrong.
> It is not designed to give you the most accurate answer, it is designed to give you the most likely series of words following your prompt.
That's not specifically true either; training is more complex than that. ChatGPT had to be trained, for example, to answer questions in a chat format.
> If an LLM is trained on 10 jackasses thinking bleach is a medicinal drink..
Again with the hypotheticals! You literally cannot discuss this subject without hallucinating things that don't exist. LLMs are trained on huge corpus of information from books to videos to reddit posts. Ultimately, statistically, it's going to predict the most common answer to something. Yes, that might be wrong but the vast majority of the time it's going to be right. And you know what, in the real non-hypothetical world, it works great. As much you don't want it to. You can hypothetically hallucinate as many weird unlikely scenarios as you want but that doesn't make it true.