Please don't imagine that I don't fully understand this.
Nevertheless, X11 "server" and "client" have confused very smart and highly technical people. I have had the entertainment of explaining it dozens of times, though rarely recently.
And honestly, still, a server is usually a remote machine in all common usage. When "the server's down", it is usually not a problem on your local machine.