logoalt Hacker News

order-mattersyesterday at 10:22 PM1 replyview on HN

how do you handle the LLM hallucinations in analysis? I like it for data extraction but i never trust it to analyze anything


Replies

eraderyesterday at 11:39 PM

First, I've spent a ton of time becoming opinionated about a normalized data model that supports the product experience I'm trying to build. This applies both to the extraction (line items, warranty sections, vendors, etc.) and the analysis portion. The latter is imperfect, but aligns philosophically with what I'm willing to stand behind. For example

- building outputs for price fairness (based on publicly available labor data)

- scope match (is vendor over/under scoping user's intent)

- risk (vendor risk, timeline, price variability, etc.)

- value (some combination of price, service, longevity, etc.)

I don't get much hallucinations in my testing, but overall it's pretty complex pipeline since it is broken down into so many steps.