> who's responsible when that clone has a bug that causes someone to make a bad trade? Who understands the edge cases? Who can debug it when it breaks in production at 3 AM?
"A computer cannot be held accountable. Therefore a computer must never make a business decision." —IBM document from 1970s
Unless not making a decision would, "through inaction, allow a human being to come to harm". — Asimov, "Runaround", 1942.
The slope between insignificant and significant actions is so enormously long and shallow, it isn't going to impede machine decision making unless some widely accepted red line is defined and institutionalized. Quickly.
If we can't agree that super-scaled predatory business models (unpermissioned or dark permissioned surveillance, corporate sharing or selling of our information, algorithmically feed/ad manipulation based on such surveillance or other conflicts of interest, knowledge appropriation without permission or compensation, predatory financial practices, ... etc.) are not acceptable, and apply oversight with practical means for making violations reliably risk-adjusted deeply unprofitable or criminally prosecuted, the decision making of machines isn't going to be impeded even when it is obviously causing great but not-yet-illegal harm.
After all, the umbrella problem is scalable harm with unchecked incentives. Ethics and accountability overall, not machines in particular.
Scaling of harm (even if the negative externalities from individual incidents seem small), has to be the redline. I.e. unethical behavior.
As a community, I think most of us are aware that the big automated bureaucracies that make up tech giant aggregators' "customer service" are already making life changing decisions, too often capriciously, and often with little recourse for those unfairly harmed.
I have personally been inflicted by that problem.
We are going to need both effective brakes, and reverse gear, to prevent this being an uncontrolled descent.
(Not being cynical. But if something is to be done, we need to address the actual scale and state of the problem. There isn't time left in human history for more slow incremental wack-a-mole efforts, or unrewarded attempts at corporate shaming. Those have failed us.)
In the hyper-scaled world, ethics mean nothing if not backed up by economics.