alt Hacker NewsIMHO (not a paper writer, but read a lot during my grad school years), the Genie is out of the bottle. The only way forward, as I see it, is using LLMs for reviews also. Basically, filter all submitted papers with an LLM and ask it to summarize it, find the biggest weaknesses and main strong points, etc. that a human can then use to review the paper. Basically, LLM-as-a-reviewer .
Personally, I would love to see a conference where people are explicitly encouraged to use LLMs for doing the work and writing the papers, and LLMs are used to review them too.
Replies
There should at least be a code of professional conduct where authors state the extent to which LLMs were used. (This would also help not wasting time by asking some “authors” about “their” paper.)
Journals themselves should make policies about the extent to which they allow the use of LLMs. In some areas it might be considered more benign than in others.
I think that’s a lot of risk of anchoring reviewer bias. I’d be more comfortable with a triaged review where the editor’s office uses models to score whether a human editor should evaluate a paper to potentially send out for review, then the editor makes their own assessment, and the reviewers continue to do their job unassisted
The entire point of an academic paper is to add to the sum of human knowledge. How can an LLM trained on a subset of human knowledge possibly even begin to accurate evaluate such a paper?
I trust an LLM to review that the language used in the paper is grammatically correct, but not to evaluate new information for accuracy.
Yes. I want reproducability, open-sourcing, accessibility, correctness, and most of all: usefulness. I don't care how it was written or reviewed, as long as some assurances regarding above things can be made, and I don't see why LLMs would get in the way of that.
What can't be gotten rid of fast enough is the notion that having written something is meaningful on its own. Making something that looks right was a level above total novice: now it's the floor.
Nope, I’ve tried this, it’s awful.
For a start LLMs love LLM generated text, so you are boosting papers people never had any input in.
Secondly, LLMs in my experience are good at small issues, but fail totally at the whole paper being obviously poorly constructed, or clearly fake.
This already happens. It's clear when a reviewer used an LLM, and it's very annoying for the authors that have to respond to what are usually low quality, superficial reviews.