I m not worried because it rewrites itself to do what? A model doesn't have intrinsic motivations, and the world destruction objective is just one point in an infinite space
Language models will absolutely have intrinsic motivations if we use extensive amounts of RL (which we do) because that heavily rewards them for steering toward future gains and losses
> the world destruction objective is just one point
No, there are quite a lot of goals that lead to world destruction. In fact, possibly most goals in that 'infinite space' would lead to such.
> A model doesn't have intrinsic motivations
How do you know that?
To quote a professor of mine, years ago:
"We don't know what grand ambitions a super intelligence might have. Perhaps it's going to be calculating the best approximation of Pi"