alt Hacker NewsWhat do you use as “ground truth”? The page says “independent sources”, and I’m sure there’s too many to list, but my question is how are they vetted as being truthful and how are two sources with opposite viewpoints reconciled?
Replies
Something I've found surprisingly effective is telling ChatGPT to "use credible sources" - you can then watch its thinking trace and see it do things like ruling out random blogs, considering media publications with a good reputation for fact checking, and double-checking information that seems unlikely.
I could imagine running multiple different AIs seeing if they agree if something is true.
I imagine this is similar to how Twitter's/X's community notes work. Something along the lines of when you have two accounts that disagree traditionally and they agree on something. That's how you know it's likely to be true.
So from that approach, you actually want to have two AIs that are on the other side of the spectrum of whatever you're trying to find out if it's true e.g. if a conservative AI and a liberal AI both agree that something is false, it's highly likely to be false.
There is no whitelist source. It can't rate sources for truthfulness. When sources conflicts - skill drops verdict to misleading or unverifiable and both are linked. Can't pick a winner at the moment. That's probably the weakest part and still a judgement call for a model.