logoalt Hacker News

bluGilltoday at 12:51 PM1 replyview on HN

All the big maintainers I know of disagree. Curl for example is getting overwhelmed with real issues that LLMs are finding. A year ago most LLM found issues where false positives (the code is correct), or hallucinations (the code in question doesn't even exist) - but that has changed and now LLMs are finding real issues in code. Many different projects have reported the same rules.


Replies

ymir_etoday at 1:26 PM

Absolutely. I'm not sure how universal this is, but I saw George Hotz tweeted something about Mythos not being such a big deal since he's able to find security issues without a claimed $20,000 in token spend per vulnerability.

The reality is that humans aren't perfect, so we do not make "perfect programs".

It is also possible to test it yourself to see how good LLMs are at finding software issues:

Find any open source project that isn't huge, then have Kimi 3 review it for potential issues.

You will find code "issues".

Most of the time it won't be detrimental for security, but often there are code paths that are not "accounted for", or you're able to bypass a type of check by interacting with the software in a strange way.

show 1 reply