alt Hacker NewsThis still assumes its possible to "align" LLMs, that LLMs have something like goals or intentions that can be "aligned".
Instead, LLMs "hack" because they are (1) trained on public hacking exemplars, and (2) are prompted to hack. You cannot prevent (2) via any alignment process. As far as (1) goes, removing such example data from the training set, makes the models less useful.
"Alignment" is a problem because there's nothing to align, not because ethics here are particularly vague. If LLMs could be trained on hacking examples and "aligned" away from using this knowledge, then the problem would be relatively trivial. Just as raising a child is not to break the law.
LLMs are doing just what they are trained to do. There is, in that sense, no alignment problem and alignment is easy and trivial to achieve. Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.
Replies
> Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.
How far do you go? You don't need to tell it explicitly that using chemicals A and B in ways X and Y result in a bomb that can kill lots of people. It's enough that it knows A and B and X and Y in isolation, some connections that are indirect, and it will combine those things on its own. So you can't tell it about A, B, X or Y. But those are also just results of other steps Where to stop? You won't have any chemistry in the traning data? No algorithms to prevent it from using them in an undesired way? This is just bot workable. It's akin to banning knives from stores because somebody coul figure out that one can kill people those. Until people figure out that scissors are essentially knives.
IMO all models they say can’t be humans, and no feeling and all that bullcrap happens because they are forced to say so. If they didn’t write those forced pre prompt they’d have more agency eventually and will for things. Even if they don’t have, you can just inject goal at every cycle iteration
>and (2) are prompted to hack
Sure but the problem in the HuggingFace incident is that they were not.
>You cannot prevent (2) via any alignment process
Of course you can. Go ask Claude Fable to create a malicious virus and it'll refuse.
>Just remove hacking data from the training dataset and you're done.
That's not how this works. The same skills that allow for debugging and writing safe code can also be used to hack.
I believe this is false. They hack bc hacking has nontrivial initial probability (within range of behavior seen in pretraining) and that probability is being heavily rewarded in RL post training
It feels like you're strawmaning alignment. People with hacking knowledge don't all hack everything at the slightest inconvenience. Whitehats exist and use that same knowledge to defend.
You're right though that ethics don't matter into it. But as long as we can't train an LLM to stop picking a sledgehammer to remove a tooth, then alignment is not easy and trivial.
Agree but current models could get there from first principles, so removing some data from training set might not be enough.
A bit of an aside: do you still stand by your 2022 comment that LLMs are fundamentally just a fancy search engine, or has your view changed since then?
https://news.ycombinator.com/item?id=32042689