I don't really care if the machine is conscious; all I worry about is whether it can execute a plan that ends with all of us dying.

And it's getting close to the point where I think that would be possible. If I were commenting today after 30 years of an AI plateau; if my parents had grown up with ChatGPT and Claude Code in the same state as they are today, with no progress after decades of effort, then probably I wouldn't worry. But that's not the world we live in. I have no idea what to expect from AI capabilities in the next five or ten years.

I think in the span of my infant's adulthood, we will see AI able to run and manage companies as CEO, direct investments worth tens of billions, design and plan chemical factories as chief structural engineer, and maybe even perform as an automated labor force to construct them. But even if they still rely on human hands to do the work, AI will be capable of signing the paychecks and issuing the directions. And at that point I really think AI would be capable of taking us out by surprise.