It's pretty widely accepted in the climbing world that the primary effect of altitude in the short-term is a reduction in your cardiovascular fitness.

The better your heart is at getting oxygen into your muscles and organs, the better it can compensate for less oxygen.

Not a bulletproof solution to altitude sickness, but it's definitely one of a lot of variables that matters. It's also just true that some people are way more susceptible regardless, I've got friends who run competitive marathon times who get splitting headaches flying from sea level to denver.