In my experience there’s a strong “banality of evil” that happens.

Some poor schlub ML Eng has shipped a feature that wins an A/B test. They’re pushing to get promoted. Their management wants to show they’re hitting their KPIs.

An engine of destruction filled with well meaning people just hoping to advance in their careers.

You might say, it’s ultimately the designers of the incentives that matter. Even there, the leadership will change. Inevitably the needs of the capitalist machine take over.