Afaik Xorg is 'only' like ~500k lines of code, which is not huge by the standards of large porjects. In fact, one of the major counter arguments against Wayland, is that the compositor + a few core libs, like wlroots necessary to provide a similar set of functionalities, is already larger that the equivalent X11 code (which has a ton of unused cruft).