alt Hacker Newsmanual PR reviews can catch things that llms currently miss
I didn't say LLMs though. I'm talking about the sort of code quality tooling that's been in use for decades - Sonarqube, Codacy, CodeClimate, etc for code quality, eslint for AST-based code rule checking (easily picks up duplication for example), Istanbul for code coverage, Wiz or Github Advanced Security for security.
These are well known industry standards for knowing a code change is correct without giving it to an LLM to guess at. If you have them, and they're well configured, and you maintain the tooling properly, then PRs become unhelpful gates that don't find problems.
I am not familiar with all the tools you listed so please correct me if i am wrong, but all these will catch stuff that LLMs can potentially catch as well (if configured correctly).
They will not handle any architectural problems (i.e. this method is correct but doesn't belong on this package, or this method is called isX - but has side effects).
And I am pretty sure that none of them do what I am saying with the tests. i.e. run tests without the associated code change and see them fail. I am also pretty certain that none of them will understand problems with breaking backward compatibility (i.e., a fix that is correct that breaks the setup of all existing users).
In other words, it is possible today to create a PR that passes all checks, all security scans, all analysis tools, all test suite while still being wrong due to architecture, backwards compatibilty, wrong scope etc. So even before and even after LLMs a human is needed there.
PRs might become unhelpful as you say if in the future one of the two things we happen
1) LLMs have unlimited context so you can pass all source code plus all architectural designs them 2) We have a super smart analysis tool that is one level above what we have today (including llms)
We are not there yet.