alt Hacker NewsThe baseline Claude prompt being compared to feels pretty laughable so not sure what is learned. Maybe compare to a more realistic baseline for the DIY side for more compelling benchmarketing?
We started with a DIY code review skill because it's inherent to want to customize to our codebase and infra before trying solutions that add layers which may get in our way here. We have a 1 page skill that that does seperate passes on security, spec conformance, proper DRY & architectural abstractions, etc, and adversarial result quality passes to prune & prioritize. Others do similar.
Once quality is fixed, I'd expect the comparisons to be less on hits/misses , and more on token efficiency. That's a tricky one bc developer local review tokens are heavily subsidized right now.
> Maybe compare to a more realistic baseline for the DIY side for more compelling benchmarketing?
This is fair critique. However, I don’t really trust myself to write a great code review skill for vLLM or OpenClaw. I also don’t think Claude Code is the right harness for this deep and broad scanning work. We find that it struggles to maintain clarity when considering many different bugs at the same time. The coding agents seem really great at single-goal tasks that they can Ralph their way to.
> We started with a DIY code review skill because it's inherent to want to customize to our codebase and infra before trying solutions that add layers which may get in our way here.
Being able to tinker deeply with the tools is pretty inherent to my love of dev tools in general. Our job is to make use of all of those customizations (our agent will use that 1 page skill when doing its bug finding). I also still think externalizing part of your dev workflow is the right way to get ahead. You really don’t want to do the work of eval-ing/maintaining that skill to make sure it still performs well with a mythos or something.
> and more on token efficiency.
I’m really confident in our ability to stretch $20 of tokens ;)