alt Hacker NewsWithout having read the article, I would say it demonstrates that benchmarks alone, without knowledge of how a model was trained (c.q. constructed), cannot reliably determine whether or not a model has certain capabilities.
Without having read the article, I would say it demonstrates that benchmarks alone, without knowledge of how a model was trained (c.q. constructed), cannot reliably determine whether or not a model has certain capabilities.
Yeah. Also, we think that in certain scenarios, it's possible for "cryptographic patterns" to arise in model weights via normal training, especially if models are trained against things such as honesty probes.