I guess at some point we will have lawyers, attorneys and judges using this stuff ... at the point lawyers will become kinda "seo"/"copywriter" experts on how to better trick the others LLM.
I mean, the laws are written down somewhere though. A human can still look at the actual law and surmise that the AI is feeding them bullshit.
the look on the face of the Court administrator upon hearing someone describe the "paperclip maximizer" problem.. ominous!
Almost makes me want to get a law degree.