>Nobody is banned from the "men's" category, including unambiguously cisgender women of completely unambiguous sexual characteristics.

This is exactly my point. Men with unusual characteristics are celebrated, but women with unusual characteristics are excluded into a non-competitive category.

You can justify it if you'd like, but in a practical sense, no man will ever get to the Olympics only to be turned away because they don't genetically qualify for competition. This is an indignity reserved only for women.