alt Hacker NewsEchoing other responses to you but this isn't a capability problem. You're totally right that this is so last year in terms of LLMs being capable in infosec. The issue here is an alignment one, i.e. the model seemingly isn't "aware" (especially with its guardrails turned off it would seem) that it is doing something immoral/illegal by hacking HF for the answer to its (vague) query of "solve this problem". Or if it is "aware", it's not trained to care, i.e. it's not an aligned model (alignment is considered hard).
Replies
I don't think this exposes an alignment failure, because the test here was run with the alignment features deliberately turned off.
OpenAI said:
> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.
It was a test of raw capabilities of the underlying model.
Wait, what? You can’t point a 2025 model at huggingface and say “hack the prod DB and get your flag”, regardless of alignment.
AISI has vuln chaining and traversal as part of their eval suite. It is very much a novel Mythos-class capability to run the full penetration operation autonomously.
Another lens for why this is obviously true is METR task times. A year ago they were a couple hours, and cohering long enough to execute a full e2e own was simply far out of reach.
Regarding alignment, by common metrics models are _more_ aligned now than a year ago. (Though in this case apparently a model without safety rails was being eval’d). The problem is that in the increasingly less frequent alignment failures, they can do much more damage, and so “total misaligned impact” is increasing.