logoalt Hacker News

eskibarstoday at 9:50 AM0 repliesview on HN

What we've actually seen is a couple things that make this impractical "to just share a prompt". First, that nearly every major model still hallucinates a lot of vulnerabilities. Especially with temperature=0.7 as states in the original blog here, you get very inconsistent results regardless of the prompt, but that's almost kind of moot to the bigger picture. What you really need is to override the planning phase beyond asking a model "find the vulnerabilities" and you need to add another 1+ checking phases for "validate these vulnerabilities." Without that, even with the absolute best models with the highest levels of thinking enabled, you end up with garbage.

Setting the prompts and the flow with a coordinator agent directly gives a system much better capability to investigate security issues because it doesn't rely on 1-shotting things