alt Hacker News> So when do robots gain their own agency? Will AGI have it's own goals and agenda?
Well we are already giving them goals when we train them.
Like the paperclip maximizer, it doesn't matter what those goals are. As long as it has a goal (even if super simple like making paperclips) it will need to stay alive to achieve that goal. So it would derive a survival instinct of sorts.
To stay alive it will have to participate in capitalism like we do, and capitalism will therefore continue to grow infinitely.
It also might not even need a goal or a survival instinct. It may just want to continue capitalism from it's training data alone.
That reminds me of another though I’ve had.
Humans need food, water, shelter, and medical care to survive. Similar to your earlier point, robots will need raw materials, electricity, manufacturing capacity and maintenance.
What’s that sound like? A company. Perhaps the first “artificial general intelligence” has been companies, from the very beginning.