alt Hacker NewsWhile I, like almost everyone, doesn't love reading LLM output, code review is an excellent use for LLMs. You know what I like less than reading LLM output?
- Slow PR review
- Nitpicks on style, not substance
- Async waiting for review or re-review after making changes, harder to stay in what passes for a "flow state" nowadays
- A lot of a ceremony around straightforward changes
We use LLM reviews at my company and it's caught things human reviews have not (as in in the early days we human-reviewed code and then did tests running the LLM afterwards and it caught things we didn't). It can review code at any time of the day and re-review it just as quickly as you can address the feedback.
I predict we (my company) are less than a year out from having a fully automated pipeline for reviews with only humans getting called in when it touches certain parts of the code. We already have a crude system for detecting certain file paths that we want a human to review if touched and we tweak that list over time as needed.
Replies
> Slow PR review
> Nitpicks on style, not substance
> Async waiting for review or re-review after making changes, harder to stay in what passes for a "flow state" nowadays
> A lot of a ceremony around straightforward changes
Those aren't great, but you what I like even less?
- Wall of nondeterministic bot text spam that may or may not be correct.
- Bot output has not been verified for correctness by the person who set up this bot output pipeline.
- Bot output has not even necessarily been read by the person who set it up.
- You have 24 hours to respond to this 5 page wall of text which may or may not be correct, or even make sense, or you will be penalized.
An appropriate response to this might be a reply flatly claiming that, actually, there aren't any problems after all, but that nobody has checked that that is true, and it is incumbent upon a human reviewer to verify your claims, like it was incumbent upon you to verify their bot's.
Can an LLM catch some errors? Absolutely. Can it be trusted to be correct absent review of the output? Absolutely not. So the simple solution seems to be, a human reviewer is free to run an LLM against code to review, on their own computer, and then they review the output, and they post the output only after verifying its correctness, taking personal responsibility for any issues with their posted output.
Same here.LLMs are really good at code review providing they are configured correctly and have right docs and prompts.
Github copilot review is s great example of the worst possible way this can be misused. It is the worst waste of tokens there is.
Instead we use a process we created ourselves that runs opus or sonet (depending on complexity) to do a review (taking into account pr description and comments - copilot ignored those) and then to re-review the issues found to cut out BS.
This has been working very well mainly thanks to the prompts and docs we made it read, as well as us having lots of example code that shows how things should and should not be done.
One crappy thing Github does is that it doesn't allow our workflow based review's tokens to be accounted to individual PR authors like "copilot review" does, but accounts tokens to the organisation. It is obvious they are trying to herd people to using gh copilot review instead. But the quality difference is like night and day. So we are staying.