Phoenix[0] has some good ideas about how X11 could be made more secure without breaking backwards compatibility. I don't understand what was so fundamentally broken about X11 as a protocol that it required a replacement protocol.
We can argue about limitations of X.org's implementation of the X server, but, as demonstrated by Phoenix, X.org doesn't have to be the only X server implementation.
I have no idea, but my impression was that basically nobody wanted to put in work on X11 anymore.