Stupidity, greed, and straight-up evil intentions do a bunch of the work, but ultimately short-term thinking wins because it's an attractor state. The influence of the wealthy/powerful is always outsized, but attractors and common-knowledge also create a natural conspiracy that doesn't exactly have a center.
So with AI, the way the natural conspiracy works out is like this. Leaders at the top might suspect it's bullshit, but don't care, they always fail upwards anyway. Middle management at non-tech companies suspect their jobs are in trouble on some timeline, so they want to "lead a modernization drive" to bring AI to places they know don't need it, even if it's a doomed effort that basically defrauds the company owners. Junior engineers see a tough job market, want to devalue experience to compete.. decide that only AI matters, everything that came before is the old way. Owners and investors hate expensive senior engineers who don't have to bow and scrape, think they have to much power, would love to put them in their place. Senior engineers who are employed and maybe the most clear-eyed about the actual capabilities of technology see the writing on the wall.. you have to make this work even if it's handed to you in a broken state, because literally everyone is gunning for you. Those who are unemployed are looking around like well.. this is apparently the game one must play. Investors will invest in any horrible doomed thing regardless of what it is because they all think they are smarter than other investors and will get out in just in time. Owners are typically too disconnected from whatever they own, they just want to exit/retire and already mostly in the position of listening to lieutenants.
At every level for every stakeholder, once things have momentum they don't need be a healthy/earnest/noble/rational endeavor any more than the advertising or attention economy did before it. Regardless of the ethics there or the current/future state of any specific tech.. it's a huge problem when being locally rational pulls us into a state that's globally irrational
Yes, that "attractor state" you describe is what I meant by "if the changes were made I doubt they could last or be enforced given the massive incentives to revert to our broken system". The older I get and the more I learn, the less I'm willing to ascribe faults in our society to individual evils or believe in the existence of intentionally concealed conspiracies rather than just seeing systemic flaws and natural conspiracies.