alt Hacker NewsDo you have a solution for degradation in accuracy when compiling larger amounts of llm-produced text?
I am also building LLM knowledge/memory systems and I've been surprised how bad LLMs are, even SOTA models, at summarizing non-trivial input batches of text. They get things wrong, distort the underlying meaning or data, etc.
Replies
dariusmonsef • today at 12:26 AM
A constant challenge. Don't have a perfect solution for it yet, but importantly every change to any article logs who did it, what it did and the reasoning behind it. So I have enough data to work with as I continue to improve things.
Divide and conquer essentially, is what I've found so far to work best. Split things into smaller and smaller chunks to independently be verified, double-check everything, then coalesce upwards with verified summarizations. Have benchmarks for every single task and sub-task that will happen everywhere a LLM is involved, so you can measure improvements. Takes a ton more effort and tokens in the system itself obviously, but if you're not paying per token, it seems to work pretty well, albeit feels slightly over-engineered already.