alt Hacker News> human lives are the cost of making the wrong decisions ...
True.
> ... AI can't be allowed to be a decision maker
Non sequitur.
Part 2 only follows from the Part 1 if you can demonstrate that AI makes more errors, or more severe errors, than humans in the identical situation.
And for those situations, I agree with you.
But there are already many situations where the AI is making fewer errors than humans, and for those situations, it would be unethical to exclude AI (see Part 1).
Replies
Better error rates are an argument for AI into the process, but that doesn't justify making it the decision maker. Ethics concerns more than just the error rate, it also concerns the accountability, ability to appeal, due process, explainability, distribution of harm, and so on.
Ethically, we can easily justify refusing to delegate responsibility and accountability to a system that has no responsibility or accountability.
Since AI can't be held liable for mistakes, it's immaterial whether the AI makes fewer mistakes than humans.