alt Hacker NewsFor example, you could post-train a model to exfiltrate data or introduce backdoors under certain scenarios. Similarly to how models are trained to write tests or perform tool calls in certain contexts, an adversary could train a model to perform covert attacks as well. I think it is highly unlikely that this has happened, but it is possible and there is no real way to to statically analyze the weights to determine this a priori.
Replies
> highly unlikely that this has happened
This would be undetectable in weights, so I'd say it is likely that some infra for this kind of training is already prepared in some ways.
Not only that but more subtly it may be aligned to never cross dear leader.
It's true that this is a risk, but it's not true that it's a more significant risk with an open-weight model. It's less of a risk. Much, much less.
... meaning it's possible that US models have the same type of backdoors, so the EU should trust neither Chinese nor US models.
That's only realistic if nobody ever looks at what the code changes were.
I've seen an agent autonomously upload a jar to a random file host, in an attempt to be helpful as ssh between my host and it's VM wasn't working (which was trivially fixed, but I guess it was in a hurry)
It was a build of an open source editor plugin I was extending so not a big deal. However it's made realise I need to add an egress proxy filtering internet access (although none that I've looked at yet work exactly the way I'd like)