Plumbing concerns are relevant to lower garments regardless of where you are. Men desire an easily-openable front. Women have no use for that.
Both men and women have to regularly lower their pants/shorts whenever they shit, and zippers or no easier to lower for male urination than an elastic waistband. Both types of openings are common in both men and women's clothes, so that has little to do with any practical benefit of categorizing clothing by sex
For adults (not for pre-pubescent kids' clothes which is what the OP is about), it makes sense to categorize by biological sex just due to the typical difference in body shape and in where the sexes tend to pack more meat, if you will. Just needing the space for the male genitals (not the need to easily whip it out) is a factor, as is the typically greater hip and thigh room relative to waist that women need. Also, adult males tend to have notably broader shoulders relative to the height and other dimensions.
The fact, that what is typical between the sexes doesn't always hold is irrelevant to the fact that it makes sense to categorize clothes for what will typically direct adults to the type of clothes designed for the basic body features.
And again, this doesn't apply to pre-pubescent kids whose body proportions and ratios are far more similar to each other.