Humans are only kind of held accountable. If you ship a bug do you go to jail? Even a bug so bad it puts your company out of business. Would there be any legal or physical or monetary consequences at all for you, besides you lose your job?

So the accountability situation for AI seems not that different. You can fire it. Exactly the same as for humans.