His rogues never really outmuscled him - it's always been their minds which were an issue. Even Bane, the guy who broke him, did so because he outprepped him.
People like Joker, Riddler, Ra's, Hush, Catwoman, Two Face, Penguin...none of these guys are a match for him in a cage fight with his gear. They're threats of a different sort.
But sure, people like Killer Croc would be destroyed.