In practice, ld.so is a piece of glibc. If you want to build ld.so from scratch, you have to clone glibc or get a tarball of glibc, and configure and build glibc.

You could make the argument that the glibc developers should split ld.so into its own subprojects. What for? That would just bring the extra responsibility of making sure that variations in glibc version work with variations in ld.so version for whatever reason.

What would happen in practice is that they would keep their version numbers in lock step, and systems integrators would use the same version. While the upstream glibc has to go through a dance of pretending that someone cares about their independence.