I think it comes from a few things during puberty.
We're usually taller than women, some have bigger penises than others, and we compete by how far we can throw and how much we can lift or how fast we can run or how witty we are etc. etc.
Women tend to help each other and act nice, men put each other down and compete and thrive on attention.
I think it's an attention complex, and a desire to measure things to determine who's the best, rather than a size complex.
Disclaimer: This might be complete shit.
Is attraction superficial, personality wise, confidence, or a mix?