My parents and my spouse's parents were all in their late 30s having children, now we're in the same position due to infertility and now finally going through IVF. We're happy it's working but at the same time it's sad knowing they'll grow up never really knowing their grandparents.
The grandparent situation is sad af. It's also pretty sad being a mid-40s year old dad that doesn't have the energy to keep up with their kid. I pitched a little league game yesterday and it wiped me out. Also, the fact I (and you) will not know our grandchildren very well also is quite sad.
If my son has his first kid the same age I had him, I'll be in my 80s when that kid is starting little league (or that age). Then, factor in the fact that I don't know of any men in my family that have lived past 80 and it gets really grim. They were all heavy smokers and drinkers I remind myself with fingers crossed.
The most sad part for me, is I realized by delaying parenthood - I was just being selfish - and the net result is I minimized "shared time on earth" with the person I love the most. It's easy to say I wouldn't have been a good parent or I wanted X job/income first, but it's all just excuses and selfishness.