alt Hacker NewsYeah I personally don’t understand the point of AI code review tools all that much, as AI is already generating the code as well. All of these AI code review tools create so much noise, yet don’t catch the really important things.
Replies
Where I find it shines it to find inconsistencies. My readme or docs or ADR something should work like X but it finds a test where it tests something different and the test is green. Or other similar drift.
Yes, your prompt need to include to look for certain “quality” aspects you care about. But once that is there it can help find a lot of things.
It can also help in finding edge cases. It is really about the prompt.
What really important things are human reviews catching in your org?
I just feel more and more like the effort invested in manual reviews is not worth it
Code review tools are designed for less organised dev teams that don’t do PRs and mandatory human reviews already.
It is papering over a lower level of competency without having to invest in actual human oversight or real process improvement.
The noise is a huge problem, indeed. Still, a panel of review agents using models and harnesses different from the one implementing a set of changes has proven immensely useful for myself. The panel is basically an n×m matrix of agents and highly specific review prompts, i.e.:
- review for intent fulfillment: is the ticket done?
- review for correctness: race condition bugs, ...
- review for security: check against this list of sources and best practices
- review for api conformity: identify all surfaces of systems outside this codebase touched by the code changes and check against their docs
- etc. pp., same for maintainability, observability & analytics, test coverage, usage of feature flags
The matrix is sparse, so not every model is used with each of the review categories. Effort levels vary, too. The next stage does a consolidation across all findings, then another stage spins up one agent per finding and investigates the whole codebases for identical / similar instances of the finding; finally, it suggests a fix.
This works extremely well for finding deficits, but the amount of noise drives me insane, too. Lots of feedback is technically correct and "by the book", but pretty useless in practical terms – or even detrimental because the amount of code written and thus the size of the change set explodes. I'm not yet sure how to tackle this problem, any suggestions are welcome!