We need to stop using the word "Alignment" because it clearly has different meanings for different people. Some people consider it to be preventing the model from waking up and enslaving or destroying humanity a la paperclip maximizer. Some people mean it as preventing the model from assisting with bad things like hacking, cybersecurity exploits, building weapons, and other things that are aligned with that individual user's goals, but not those of society more broadly.
Wow, they really misuse "alignment" in the article! They give a decent definition, and then proceed to ignore it.
Alignment == "training an AI system to act in a manner that aligns with human interests"
...but then they talk about guardrails and permissions as part of alignment. I think this is their weakest argument
What “alignment” refers to is the ability to robustly convince a model to achieve anyone’s goals. This is really quite obvious and nearly everyone intuitively recognizes it; you use the word “aligned” this way yourself!
The source of the debate is that some people have a strong preconception that alignment can’t be a hard problem. GGP takes it for granted that alignment is solved and anyone who wants to can build a system that won’t cause harm. So picking a new word won’t help.
Those are, unfortunately, much the same thing.
The paperclip optimiser thought experiment isn't "AI does this spontaneously", it is "AI is told to do this by someone who didn't realise the consequences, and it succeeds".