alt Hacker NewsIt seems possible to deliberately not train on some offensive capabilities and still have a very useful model. For example, Opus 5 deliberately did not train on exploiting vulnerabilities, and so performed less well on exploits than Mythos, yet was equally proficient at finding such vulnerabilities, according to the Opus 5 system card in their "OSS-Fuzz" eval [1].
[1]: https://www.securityweek.com/anthropics-opus-5-nears-mythos-...
Replies
jbstack • yesterday at 10:33 PM
It's hard to imagine a frontier model being proficient at finding vulnerabilities, but not so proficient at exploiting them.
Surely finding is the hard part, and any LLM should be able to easily exploit a vulnerability it already knows about?
➕ show 1 reply
That's a good example, but I'm unsettled by Anthropic's growing refusals in the areas of chemistry and biology. If they think that scientific assistant models should be as unhelpful as Fable, because applied scientific knowledge is inherently dangerous, I don't want Anthropic or like-minded thinkers setting the standards for model safety.