Or in one of my favorite coinages: "Moral Crumple Zones". "Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions." See https://ferd.ca/notes/paper-moral-crumple-zones.html

Yep and the car wasn't self-driving at the moment it crashed.

If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company.

Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it.

This is not a new concept and it’s not an idea the LLM companies invented. It shouldn’t be surprising to anyone.