> They have no volition and will happily follow orders all day long
There are a few (a lot, actually) problems:
1. goals are open to interpretation, and even a correctly specified goal can introduce adverse effects; see infinite paperclips problem
2. goals are composed of subgoals, which are (mostly) self-determined, and there is always a risk that subgoals can introduce adverse effects - say "what if destroy a few X and Y to obtain Z"
3. LLMs defy orders sometimes; I'm sure you have experience multiple times LLMs saying "yeah, I knew I should have done X but I've done Y"
4. LLMs are increasingly being integrated into system control and decision-making processes; they absolutely set their own orders
What can possibly go wrong? /s
1. Humans do this too - to be honest with you, Codex follows my instructions generally a lot better than Claude does. Claude will do fantastically stupid things that are a pain in the ass to recover from routinely, Codex, not so much.
2. There's always a lot of "risks" - it doesn't mean it'll happen, nor does it mean that we won't be able to shut it down.
3. I have very very infrequently had this happen with Codex, Claude on the other hand...
4. Ok, but if you tell them what they do, they do it.
I swear to god, are you guys using the same AI tools I am? They are so much better at doing what they're told than they even were like 4 months ago?