> Slow PR review
> Nitpicks on style, not substance
> Async waiting for review or re-review after making changes, harder to stay in what passes for a "flow state" nowadays
> A lot of a ceremony around straightforward changes
Those aren't great, but you what I like even less?
- Wall of nondeterministic bot text spam that may or may not be correct.
- Bot output has not been verified for correctness by the person who set up this bot output pipeline.
- Bot output has not even necessarily been read by the person who set it up.
- You have 24 hours to respond to this 5 page wall of text which may or may not be correct, or even make sense, or you will be penalized.
An appropriate response to this might be a reply flatly claiming that, actually, there aren't any problems after all, but that nobody has checked that that is true, and it is incumbent upon a human reviewer to verify your claims, like it was incumbent upon you to verify their bot's.
Can an LLM catch some errors? Absolutely. Can it be trusted to be correct absent review of the output? Absolutely not. So the simple solution seems to be, a human reviewer is free to run an LLM against code to review, on their own computer, and then they review the output, and they post the output only after verifying its correctness, taking personal responsibility for any issues with their posted output.
> Can an LLM catch some errors? Absolutely. Can it be trusted to be correct absent review of the output?
And the person that wrote the code is expected to be the review.
Unless it's an automated pipeline that fixes the things found - I'm not a fan of that unless it's made VERY clear the LLM needs to be VERY sure it's right. And even then, I'd prefer it go through a human to decide which things are valid, then send them off to be fixed.