You can always reduce the number of jobs from make/ninja to 4...
But, yes, you are right; the issue lies on linking. LibTD requires 2GB as minimum, but linking takes ages. At least with GCC, Clang requires far less RAM.