alt Hacker NewsA 0% false-positive rate is not necessary for LLM-powered security review to be a big deal. It was worthless a few months ago, when the models were terrible at actually finding vulnerabilities and so basically all the reports were confabulated, with a false positive rate of >95%. Nowadays things are much better - see e.g. [1] by a kernel maintainer.
Another way to see this is that you mentioned "LLM found this serious bug in Firefox", but the actual number in that Mozilla report [2] was 14 high-severity bugs, and 90 minor ones. However you look at it, it's an impressive result for a security audit, and I dount that the Antropic team had to manually filter out hundreds-to-thousands of false-positives to produce it.
They did have to manually write minimal exploits for each bug, because Opus was bad at it[3]. This is a problem that Mythos doesn't have. With access to Mythos, to repeat the same audit, you'd likely just need to make the model itself write all the exploits, which incidentally would also filter out a lot of the false positives. I think the hype is mostly justified.
[1] https://lwn.net/Articles/1065620/
[2] https://blog.mozilla.org/en/firefox/hardening-firefox-anthro...
> A 0% false-positive rate is not necessary
To be clear, I'm not saying 0% false-positive because that will always be impossible with any LLM.
However, to greatly over-simplify what I already said ...
The presence of >0 false-positives means you still need someone who knows what they are doing behind the keyboard.
The presence of an LLM, no matter how good, will never remove the need for a human with domain expertise in security analysis.
You cannot blindly fix stuff just because the LLM says it needs fixing.
You cannot report stuff just because the LLM says it needs reporting.
There may well be scope for LLM-assisted workflows, but WHO is being assisted is a critical part of the equation.
That is the fundamental point I am making.