I don't think speed would be any major advantage. Marvel doesn't use it often in combat. At least not on the level it would give him any advantage over Nefaria.
Original Nefaria - He might have the raw physical edge but he was a bit of a pansy and wasn't very experienced. The lack of flight was also a disadvantage. Captain Marvel would edge him out.
Busiek's Count Nefaria - Captain Marvel edges him out once more. Count Nefaria might be experienced and ruthless at this point, but I'd say physically he was weaker than his original incarnation.
When Thor hit him with the hammer, and then with his right fist, Nefaria was basically thinking "That... hurt! But not as much as I'd have thought! Thor should fear me!"
Not word for word, but that's the jist of it. The fact he wasn't stunned and still able to fight on is what I meant by "shrug off".. He took Thor's opening shots, and gave as good as he got.
Shrugged off doesn't mean he didn't feel the attack, or get hurt by it. It simply means he took his best, and was still in fighting form.
Although, to be fair, that likely wasn't Thor's best.. Not even close. But it's still about as much power as Thor usually uses to defeat class 100 opponents.
He releases his eye beams -which Thor blocks- and then tosses a Skyscraper on top of him - which Thor promptly digs himself out of. He then stops a hammer strike and knocks Thor on his butt while he was caught unaware and off balance.
In my book it is. In that arc Count Nefaria was shrugging off Simon's attacks. But every time Thor landed a blow, he was rocked.