It's one of the relevant factors. It, and related facts, make it usually possible for a man to overpower a woman (and a predator self-selected for being somewhat above average in fighting ability might be confident of overpowering multiple women, or at least being able to get away in the worst case), which has implications for safety.
Women-only gyms are because women don't like being oogled whilst exercising.
Maybe some claim it's for safety but the fact they're often 24hr would decry that.