A few current situations that are leaning this way in theme:

- ai facial recognition used by police, detaining innocent people with no recourse or consequences

- ai military decisions made without human in the loop. Double points for the decisions being to kill someone. Anthropic insisting a human should be in the loop for killing decisions is what caused Trump to declare them a supply chain risk.

- ai denial of insurance claims without a doctor in the loop

- ai "plagiarism" detection in college courses failing students

- that one colleague everyone has who throws slop over the wall and just sends any feedback directly to the ai

The thing you mentioned, human judges and harsh penalties for unsupervised ai lawyering, is trying to hold this kind of nightmare back. It will be very hard (and only get harder) for humans to fight through the deluge of slop, especially if the slop is weaponized as a kind of DoS like in the book. I don't expect laws are strong enough to hold this back but I don't know any other tool in our collective toolboxes.