alt Hacker NewsOf course this is authored by people at Anthropic. Continuing their narrative that open sourced models are dangerous and that people should trust the closed sourced providers.
This of course is not an issue if you actually care about security where you would've already sandboxed your agent and put guardrails on the inputs/outputs of the model.
> Poisoning only 0.1% of the pretraining corpus
Considering the scale of the datasets used by most companies, being able to poison one out of every thousand documents used in it is a huge leap.
Then, on top of that, you have to also then get your trigger somehow sent to an agent that will honor it.
I guess I'm just not impressed? Like yeah, if you train a model to do bad things it will do bad things, that shouldn't be shocking to people.