alt Hacker NewsNo; it’s a signal of lobotomy. Models get dumber near where you sensor them. You could almost think of it like your prompt is getting redacted, and so the LLM has to deal with these holes in your prompt (where you mentioned the semantic contraband) and it doesn’t do a very good job because it has no idea what you’re talking about.
The mechanism is likely a bit inverted, it’s the weights that have been redacted, but the effect is that weights that were interfered with are affecting the problem solving abilities of the LLM - I mean, it’s insecure bad code, not a backdoor, right?
Replies
I would have to clarify the test mechanisms of crowdstrike. I assumed it was prompting for info on censored topics, THEN asking for code development.
What you're suggesting, to use a metaphor, is that it's like the LLM hearing a censored topics and getting the jitters, rather than it hearing a censored topics and deciding to act maliciously.
"Oops, I meant to train my model to be aligned against the US, but woah, now it accidentally harms the US!! My bad!!"
Not convincing, sorry.
Six of one and half-dozen of the other.Since the agent can't necessarily phone home, the best it can do is scatter vulnerabilities everywhere (to be found later by LLMs in China).
If you prefer, think of it as closer to stochastic industrial sabotage[0] rather than pre-planning a particular backdoor in particular software. Different tools, different tactics!
[0] https://news.ycombinator.com/item?id=35448090