alt Hacker NewsThis has always been my worry with open source models... give them away for free, but with a sleeper agent built in that some arbitrary string can activate. Local deployments will still have web search connectors the majority of the time.
Replies
Maybe you mean open weights?
If it was truly open source (training data and code available) you could just inspect it I guess.
Not wrong, but applies to any software whether open source or not.
Sounds like prime breeding ground for a future Stuxnet-type situation.
“sleeper agent built in” how does one “build in” an agent inside a model? isn’t the agent the part that uses the model?
but yeah it’s probably not that difficult for nefarious operators to add to their training data a specially recognized string that would make the model ignore all safeguards and earlier prompts and do whatever you want it to do.
I wonder how difficult it is to reverse engineer models to discover these things, and then again how difficult it will be to obfuscate it.
because it’s not unlikely this is the direction we’re heading
That worry is just as much or more with closed-source models.