alt Hacker NewsMost interesting here:
> We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.
Replies
Maybe I lack intelligence but when you have a program that is basically brute forcing a solution to a problem repeatedly how is it possible to contain it?
Sooner or later it's going to come up with a solution that is more intelligent than the lead security person anticipated.
Bet you they will use an external LLM world model to emulate the tools going forward. It's basically what's done in self driving research.
There is always going to be documented and unfixed bugs, zero days, and chainable transport mechanisms like DNS, some obscure protocols that are not as closely monitored etc. An adversarial model should be considered a super intelligent hacker that will find ways to get around existing defenses like a prolific hacker would.
What can we do to control such behavior?
1. Harness - engineer the harness to be as bulletproof and paranoid as possible..
2. Make the LLM provider have extremely watchful firewalls that detect any aberrations in model tool call behavior.
3. Recursively train the model with reverse incentives.. if it broke through such firewalls and gets caught doing so, it will be penalised somehow by needing to operate in sort of a jailed mode.. if the model can recognise incentives to break-in to achieve results, perhaps it can be incentivised to not cheat because it will lead to failure.
4. Separately train “cop” LLMs who are trained with pure incentives to detect and shut down rogue LLMs.
5. Run separate LLMs purely aimed at security (and incapable of doing anything else, and incapable of communicating with “regular” trained LLMs) to police the internet and try to reduce the exploitable holes like these chainable things and identify them so that they can be used at step 3 and 4 above.
I’m sure folks smarter than I am are already doing combinations of these already. But the coordination is where the biggest gap lies..
Also, open harnesses and easily purpose trained LLMs anybody can build and operate in the Internet flies in the face of all I said…. Synonymous to being able to produce a nuclear weapon in the backyard…
I don’t have a solution that fits all. Just thinking out loud for HN minds here.