alt Hacker NewsSame here.LLMs are really good at code review providing they are configured correctly and have right docs and prompts.
Github copilot review is s great example of the worst possible way this can be misused. It is the worst waste of tokens there is.
Instead we use a process we created ourselves that runs opus or sonet (depending on complexity) to do a review (taking into account pr description and comments - copilot ignored those) and then to re-review the issues found to cut out BS.
This has been working very well mainly thanks to the prompts and docs we made it read, as well as us having lots of example code that shows how things should and should not be done.
One crappy thing Github does is that it doesn't allow our workflow based review's tokens to be accounted to individual PR authors like "copilot review" does, but accounts tokens to the organisation. It is obvious they are trying to herd people to using gh copilot review instead. But the quality difference is like night and day. So we are staying.
Agreed. It's sad how bad GH Copilot is, and thank you for reminding me to cancel that subscription. I just realized I haven't used it in a while now.
We also settled on a custom solution for reviews. We started with reviewers that claude/codex on GH actions and those were "ok" but kind of dumb and hard to tweak (GH Actions is just miserable to debug/iterate on). We still use them occasionally (@tag them to kick off a review) but we've shifted to a piece of software that we wrote that watches GH PRs and once certain labels have been added and the ticket in our ticket tracker (not GH issues) is marked as "Ready for Review" it kicks off a review and it reviews _all_ the PRs connected to the ticket. We don't have a monorepo, yet, so a new feature might span 3-4 repos (server, client1, client2, ...) and we want that work reviewed together.
We are close to having our review code spin up a fully isolated devstack running all our, many, services so it can poke around in the UI and write custom playwright scripts to test the new functionality. We have the devstack VMs working, just gluing the last of it together, it's pretty cool to see it all come together.