alt Hacker News...I know, which is why "as a Language Model and not a real doctor" is a pointless comment to start off with. It should simply not recommend treatment if it's not sure it is correct. I wouldn't blame it or anyone if they asked for help treating a stiff neck, and the LLM (or your neighbor or parent or spouse) suggested light exercises to help relieve it - and do not jump to the suspicion that you may have meningitis.
As a Human, I do not need to know it is a Language Model.
Replies
Yeah it should just state thing it means. But then again, there’s psychological impacts on society that we must be careful. For instance, teenagers talking to AI. If the AI just talks, people already start to feel real connections to the seemingly human entity. Maybe it’s better to disclose the reality up front?
> It should simply not recommend treatment if it's not sure it is correct
Oh okay, darn, guess they just forgot to make it so!
LLMs are famously bad at determining "if it's not sure it is correct." They are always confident, because a confident tone ranks better in RL.