logoalt Hacker News

aster0idtoday at 3:21 AM4 repliesview on HN

How many false positives did the AI throw up?


Replies

b1temytoday at 5:51 AM

Even if it does have false positives, I expect it would make a nicer starting point for finding and verifying bugs/vulnerabilities, compared to wading through the entire codebase until you find something. Even if it is a false positive, it would probably be due to sketchy looking code (hopefully, unless it hallucinated completely new code) that you can take a look at, and maybe spot something else that the AI didn't catch.

Besides the HN submission, XBOW and Hacktron AI has found plenty of vulnerabilities in code.

tyretoday at 3:31 AM

Does it matter? They found 12 vulnerabilities. Clearly there was enough signal:noise that they could uncover these as real.

It doesn't look like they had 1 AI run for 20 minutes and then 30 humans sift through for weeks.

show 2 replies
awesome_dudetoday at 3:38 AM

They don't appear to go into detail about anything except how great it is that they found the bugs, what those bugs were, and how rare it is for other people to find bugs.

I think that it would be helpful from a research point of view to know what sort of noise their AI tool is generating, but, because they appear to be trying to sell the service, they don't want you to know how many dev months you will lose chasing issues that amount to nothing.

ape4today at 3:30 AM

I wonder too. Did it take many human hours to verify everything?