alt Hacker News> to conduct effective evals, you sometimes need the model to have (restricted) access to the internet
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
This particular evaluation, maybe not, the problem is a huge amount of the evals do need internet access. There are seemingly tons of evals that require internet access as you cannot get a rich enough simulated environment. AI can detect these lower entropy environments and act accordingly when it's being tested. That should make everyone think twice about what we're building.