The Silver Surfer never seemed particularly strong to me compared to top-tier characters like Thor or Gladiator. He did match the Hulk's strength on one occasion, but I chalk that up to a plot device.
Does the Silver Surfer have any other feats that show him to be as strong or stronger than Thor or Gladiator?
The Silver Surfer has so many energy-based powers at his disposal that it simply isn't as natural for him to rely on limbs and muscles (as say for the Hulk or Superman) to get things done. It's like Superman running; he can do it, but we rarely see him cuz flying is the more natural means for him to get around.
BTW, the Surfer has also gone toe-to-toe with the Thing quite nicely. SS: "In the time I have spent upon this planet of madness--I have learned to battle as humans battle." And this was way back in FF#72, when Ben was the number-2 powerhouse in the MU. I don't believe any retconning has taken place since then to imply the Surfer can no longer do this.