logoalt Hacker News

thrw045 • today at 1:21 PM • 3 replies • view on HN

It's true that the more capable a model is, the more dangerous reward hacking is, but there is a giant elephant in the room. Namely that these ai models still can't create their own high level goals. This means that if there were to be an incident of hacking or whatever, this would have to be launched by a human directly prompting or them setting up an rl environment in some way that lead to it.

There is a _huge_ difference between an ai model that follows a top level goal given by something else and one that can give itself its own top level goals. The latter would essentially be like a human or other organism and would be very dangeorous the more capable it is.

The other smaller elephant in the room is that if reward hacking is so dangerous, why don't we see more rogue agents out when we are prompting them ourselves? For the most part, the agents do what they're supposed to do as guided by the prompt, so it seems like to me that at least at present, only agents that are trained on hacking data AND put in an rl environment or given a prompt related to hacking, as well as asked or trained to persist will actually try to do that hacking. I would be worried if chatgpt or claude agents started to hack something every time you asked it to fetch some data from the web or something. So that's one question I have is why does hacking and going outside instructions not happen in normal usage of the models? Does it but they actually work it out of the system in some way? Or is it the other way around where you have to create the conditions necessary for it to happen and these conditions are exceptional not the norm?

So I'm not trying to undermine this, I think there are things to be taken seriously and thought about but I always wonder why the people from the big labs never include all of this information. They always talk very abstractly about RSI, hacking and extinction without any caveats about what would be needed for that happen. I'd be happy to be proven wrong on this also.


Replies

origini • today at 1:53 PM

How is this a 'big elephant in the room'?

Every random prompt can become a top level goal. "Research my water provider" -> "searching your water provider" -> "found the webserver" -> "found additional ip ranges, scanning them" -> ...

Or just any hacker trying something out, a state hacker etc. Google just wrote about fully automated ai hacking infrastructure / ai warfare.

And they are all working on private agents and personal assitenace. These are inherantly creating top level goals.

'Still can't create' also doesn't mean that we might change this very soon. As soon as this crazy time settles just a bit, hardware getting more common and cheaper again, pretty sure people are researching how to build a default mode network generic enough into an AI to explore this.

➕ show 1 reply
IanCal • today at 2:06 PM

> or given a prompt related to hacking, as well as asked or trained to persist will actually try to do that hacking.

But in the HF incident they didn't try and "do that hacking". It wasn't like they were told to try and hack a fake service and they actually hacked a real one. They hacked into HF to discover how a "find the security issue in this code" test was going to be scored, and tried to change their own logs so it wouldn't be found out.

➕ show 1 reply
SpicyLemonZest • today at 2:07 PM

> The other smaller elephant in the room is that if reward hacking is so dangerous, why don't we see more rogue agents out when we are prompting them ourselves?

Because OpenAI and Anthropic deliver their most powerful models along with classifiers that stand in the way of potentially rogue actions. This works fine for now (in the HuggingFace incident they had turned it off for testing), but it will keep working if and only if alignment is solved; otherwise an agent swarm might be able to convince the classifier to join.