Your're missing my efficiency angle- it's not that the people aren't there, it's just large nations are not as good at small nations at finding them.
Plus, as I mention, Europeans enter into more events.
Despite not having a full-blooded cultural interest in athletics, the US still dominates that sport by a wide margin. Try adding up the athletics medals in the 2012 Olympics and see how Europe does then- you'll see the US wins that pretty well.
But take Equestrianism. GB won 5 medals in that in 2012. The US won 0. Is this because of some innate inferiority of the US? No, it's because the US doesn't give a damn about equestrian sports. There's no culture there about being good at it. There are several sports like that in the Olympics which various European cultures go for and the US likely never will.