Surely the only rational reason to care about liability is you get paid in the bad outcome.

Let's assume you get paid 0$ for AI mistakes -- no liability at all. Then your EV is roughly:

(AI accuracy - human accuracy) * (cost of accident) - (1 - human accuracy) * (human payout)

Now I don't know about you, but even with pretty big numbers on the human payout scale and relatively small differences, I would not accept 10 million (presumably paid to my next of kin) to die. Does lead to weird situations where I may prefer humans for minor medical procedures where I can take the risk -- but only in edge cases.

Or is there some weird punishment thing going on here?