Part of it is the uncomfortable fact that it is not very much fun to spend time with people who have small children -- it's hard to have a real conversation for more than a few minutes, and if you yourself don't have kids, what their life revolves around right now (daycare, finding a kindergarten, etc) is just not going to be very interesting.
> the uncomfortable fact it is not very much fun to spend time with people who have small children
This is (obviously?) not a fact. I’ve had a blast hanging out with a family in Peru for the last 24 hours. I also always have a great time when I visit my sister, her husband and their little kid.
Long conversations about interesting topics are one way to have fun. And you’re right, those don’t happen as much if you just take a child free couple, some parents and maybe kids and put them in the same room.
Bigger get togethers help a lot. One kid is a handful, but six kids of varying ages can actually be easier. You can also have fun in other ways like dancing, decorating or lighting fireworks (one activity from last night).