logoalt Hacker News

jostmey06/16/20254 repliesview on HN

This only addresses a small part of the problem with peer-review. The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers. If the paper doesn’t fit what is expected, it will often be rejected. This can also lead to self-reinforcing views that ignore contrarian data. Also, the data can be made up, and if it makes sense to the reviewers, it is generally not questioned


Replies

kergonath06/16/2025

> The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers.

This is a misunderstanding of the role of peer review. The point is not to prove that a paper is correct, the point is to ensure a minimum level of quality. You are entirely right that most reviewers cannot hope to reproduce the results presented, and very often for very good reasons. If I write 3 proposals over the course of a year to get some beam line on a neutron source, it is completely unrealistic to expect a referee to have the same level of commitment.

I think this hints at a more profound problem, which is that a lot of studies are not replicated. This is where the robustness of a scientific result comes from: anybody can make the same observation and reach the same conclusions under the same conditions. This is the real test, not whether you convinced 3 or 5 referees.

The real value of an article is not in whether it was peer reviewed (though the absence of peer review is a red flag). Instead, it is in whether different people confirmed its main results over the years that follow publication.

show 1 reply
yummypaint06/16/2025

Have you personally reviewed a non-zero number of papers? What is this statement based on? For a thread ostensibly about science, the comments are disappointingly lacking in evidence and heavy on vibes.

Maybe people could learn about what peer review is before posting their strong feelings about it? The purpose certainly isn't to replicate people's experiments, that happens after publication and not by referees. One of a reviewer's duties is to look at whether the study could be replicated given the included information. That is a very different thing.

Also, just because something has made it past peer review also doesn't mean it isn't controversial in the field.

show 2 replies
aDyslecticCrow06/16/2025

Bad quality work is a much bigger problem than dishonest work. Systematically well-done research with fake results or data is much rarer than just... lazy bad science.

There is a massive incentive to publish. Inflate the value, inflate the results, and stretch out projects to multiple smaller papers, fake results to make it seem important. This is lazy and fast, and can be caught by a stricter review and scrutiny.

Papers that are properly done all the way through, but with faked data meant to push an agenda, can be disproven by counter research.

show 1 reply
jxjnskkzxxhx06/16/2025

Reading your comment makes me think that you believe that the point of peer-review is to ensure that a paper is correct, or at least that specific aspects of it are correct. Is that the case? What do you think the point of peer-review is?

show 1 reply