alt Hacker NewsThanks John. Neat to see you on the HN front page.
One LLM feature I've been trying to teach Alltrna is scraping out data from supplemental tables (or the figures themselves) and regraphing them to see if we come to the same conclusions as the authors.
LLMs can be overly credulous with the authors' claims, but finding the real data and analysis methods is too time consuming. Perhaps Claude with the right connectors can shorten that.
Thanks. I can guess who this is but not 100% sure.
Totally agree with what you're saying. This tool ignores supplemental materials right now. There are a few reasons - some demographic, some technical. Anything that smells like data science would need more rigor.
Have you looked into DocETl (https://www.docetl.org/)? I could imagine a paper pipeline that was tuned to extract conclusions, methods, and supplemental data into separate streams that tried to recapitulate results. Then an LLM would act as the judge.