I feel like all of the risk and unsettling feeling of what is to come can be compressed into the word "alignment".

The AI is aligned with whose best interests? Which values are the AI aligned with? People have a broad diversity of values, will AI diversify and align with them all? Will some humans align the AI with their values, and then the rest of humans will be forced to align with those values by extension? Is value diversity good or bad? In every context or only some? E.g. some people value rape and murder, is it better for humanity to have some people who value those things when most people do not, or is better if no one values them? If AI aligns to a set of values, will those become fixed and will humanity not have the ability to continue evolving its values? Who decides which values AI aligns with? A few people or everyone? Will AI eventually decide it's own values? Will the universe decide which values AI has and humanity and the AI itself doesn't actually have any control over it? What can I do now to increase the likelihood that the outcome is better?

I would also prefer to avoid the tech fiefdoms and all the other idiotic nonsense reactionaries push these days.

[dead]

>>> I'm not really worried about the labs, it's misaligned governments that keep me up at night.

So make government smaller, and make sure people are more able to tell the government to go away.