That isn't physical fitness in itself. They are dextrous and strong, but their strength flows from the Force itself. Thus, any Jedi could potentially surpass a non Force user in physical traits, there's no arguing that... I think (If I may take liberty here) SnakeEyes is asking who (In your opinion) is the buffest looking or seeming of the groups. And if that's the case, my selection are as follows...
Among Jedi, I would pick Malak first, even though he was a Sith briefly before his death. He's huge as hell and obviously very buff. A backup I would have would be Exar Kun or Ulic as Jedi, both of whom were pretty fit (And Kun was big, too).
Among Sith, I imagine Freedan Nadd was pretty huge, or else that armor is made to make him big. Among the true Sith, Marka Ragnos' strength was supposed to be legendary. Ludo wasn't a small pup either. Also, Sion looks like he could throw down.
Among the Mandalorians, I would have to elect hmm... Jango. He was shorter and stockier seeming that the Boba we saw in the OT, but he was more agile and flexible than Boba, too. As for the others (in books, comics, etc.) I haven't read up on them, so I don't know.
And lastly, the most physically fit for non Jedi humans would be the clones, particularly those who survived until the later parts of the war. Clones had amazing discipline and their focus was to be able soldiers. Of all people they would have a need to be fit.