> The xserver (x11 or what as its old name)

It was XFree86 until around mid 00s after which the X.org fork took over.