The trade offs are real, but a suppressed desire to have kids can lead to dissatisfaction and with time that will be hard to ignore or fix.
Also, let's just not ignore that Musk is wildly successful in having shown you an example that you don't have to choose one thing
He sure as heck failed at the family thing.