I feel like the right legal solution is to make the service providers liable in the same way if you offered a service where you got diagnosed by a human and they fucked up, the service is liable. And real liability, with developers and execs going to jail or fined heavily.

The AI models are just tools, but the providers who offer them are not just providing a tool.

This also means if you run the model locally, you're the one liable. I think this makes the most sense and is fairly simple to draw a line.