I think the issue with these kind of stances is that they are basically status quo bias; why don't you object to the computer itself, and thus refuse to write programs? After all: they were invented by the UK military in the pursuit of military goals (and much of their subsequent development was funded by the US military - see https://types.pl/@graydon/110648447694201698 - and the fact that ARPAnet, GPS, etc were all military creations). Computer systems are mostly used by large corporations and the military to achieve their goals more effectively.
Usually the objection is that "oh well, the computer can be used for many great things", which isn't particularly satisfying because, um, we can use AI for "good" (better?) things as well (e.g. trying to find novel cures, unlocking the mysteries of protein folding, etc etc).
Then the objection becomes something like "well the computer is here and we have to live with it", which is also now true of AI. Do I like the "it's inevitable" argument; no, but it's clearly very true that we do have the transformer, that won't go away - where we DO have control (or should seek to change) is the organisational structures that we as a society decide to create, and how we safeguard the dignity of the individual in changing times.
Being able to discern what is and isn't in our control helps tremendously in doing what is right and constructive.
The fact that some people opt out of engaging with AI, I think is healthy for society as a whole. If that's within their control and they exercise their control to do what they think is right, then I commend them.
That said, I do think there is a greater natural force at play, something involving entropy and increasing complexity and energy profit maximization. It seems to cut through all levels of abstraction from organic chemistry to civilizations and probably beyond. I assume this is outside of humanity's control, and therefore outside of any individuals control.
So what is inside our control? Our own perceptions and actions.
My perception is that the advance of computation and by extension proliferation of probabilistic programs (AI) is inevitable. It's on a continuum that is a force of nature.
What I might have some control over is choosing to harness that potential to increase future prosperity for more people and the greater environment, and to avoid contributing to outcomes that harm people and the environment.
Lots of bad things are happening and will happen that are outside my control.
I do genuinely believe that the capabilities are inherently neutral. Civilization can choose to harness them in a variety of ways, for a variety of purposes.
If the majority of people choose options that are game theory win-win, then the future will be better... If the majority of people choose win-lose, then the future will probably be worse.
The risk isn't AI, it's how we choose to use it.