Well, at least they don't expose themselves to civil liability by doing so. The system is generally designed so that judges don't have to answer to the participants in the cases they are presiding over. That doesn't mean they don't answer to anyone.
That's the key point here. You won't be able to sue a judge who does that, but he could still be impeached. And probably would be.
You would probably have a really good shot at an appeal, too.
In U.S. history only 8 federal judges have been convicted and removed. In a normal year the number is zero, out of ~870. For state judges, about 11 are removed per year out of ~30,000. They can be held accountable, but it takes something extraordinary, and worst case they retire early. If a judge confesses to full self-replacement that might do it. Too many load bearing emdashes wouldn't be enough.