I feel like in either case any opinion should have to depend on how the group explians the purpose of it's policy, and has to actually be merely a "social group" with no material consequences to membership.

People seem to have an opinion that reflects the nature of both points here.

There are various reasons that female only groups might need protection that don’t apply to males. So yes you might expect different views.

Yes. It's called a power imbalance. The two things are not equivalent for that reason.