The other key element with Wayland is that the kernel does a ton of the work for you. There s GEM buffer management and DMA-BUF to manage and move around video & regular memory, there's kernel mode setting, there's incredibly good mesa drivers.

X didn't have any of that to build from. It basically was a second kernel, was the OS that dealt with the video card atop the OS actual. It talked to the PCI device & did everything.

Part of the glory of Wayland is that we have fantastic really good OS abstractions & drivers. When we go to make a display server, we start at such a different level now. Trying to layer X's abstractions atop is messy & ugly & painful, because it mostly inhibits devs from being able to use the hardware in neat efficient clean direct modern ways. You have to write an X extension that coexists with a patchwork of other extensions that slots into the X way, that can figure out how to leverage the hardware. With Wayland, most compositors just use the kernel objects. There's much less intermediary, much less cruft, much less wild indirection & accretion to cut a path for.

And as you beautifully state, competing libraries can decide what abstractions & techniques work for them. There's an ecosystem of ideas, a flux to optimize hone & improve, on a variety of different dimensions. The Bazaar free to find its way vs the one giant ancient Cathedral. It's just so so so good we're finally not all trapped inside.

Tl;dr: Wayland has a much higher level that it can start from. And trying to use gpu's & hardware well in X was a nightmare because X has a sea of abstractions - extensions that you had to target & develop around, making development in X a worst of both worlds low level but having to cope with a so many high level constructs you had to navigate through.