The power of the tool itself will be overshadowed by the motivations of its real owner. I can be both impressed by its ability to empower me, and be scared of the fact that the tools will change hands sooner or later and be deployed at scale to serve a goal I cannot, at minimum, support.
When most engineers and Marvel fans watched Tony Stark in Avengers collaborating with Jarvis they thought of Jarvis like "an AI with Google's knowledge where I can interact with him". It's true that we're close to that level interaction. However, the ultimate goal is to get as much as possible automated on Jarvis, to the point where Tony Stark is not needed or Tony Stark can be replaced by anyone with a mouth.
In this example, Jarvis isn't the goal but a checkpoint. The goal is a genie, providing software and research to anyone who is loaded with money, and knows how to rub the metaphorical lamp the right way.
> the tools will change hands sooner or later and be deployed at scale to serve a goal I cannot, at minimum, support
Personally, the tools don't need to change hands at all. They are already in the hands of people who are deploying them at a scale to serve goals I cannot and do not support
The people running AI companies right now are some of the most evil motherfuckers on the planet
> The power of the tool itself will be overshadowed by the motivations of its real owner.
Not only that, but by how blatantly and openly these owners are discussing the tool's power. They are publicly crooning about their product's ability to replace workers. It's the first line of their sales pitch. And also, their customers (business CEOs) are publicly crooning about how awesome it is that they can reduce their headcount! Both the AI producers and their customers are absolutely bragging about worker displacement, and not a single guillotine has been constructed in the streets yet.