Kingpin is below either DD or Punisher IMO, who are more comparable to Ward in terms of skill and ability (though I do put Matt a bit above the other 2). Season 1 Daredevil was not as strong or skilled as he is in season 2. And he wasn't totally recovered from his fight with Nobu during the finale either. And then there is the fact that Fisk was fighting to kill whereas Matt was fighting to capture. People try to bring up the season 2 jail scene where Fisk knocked Matt around during their conversation as evidence that Kingpin is somehow superior, but ignore the fact that DD was in his public persona, and couldn't really fight back without revealing himself. And Castle was not only battered, but shackled when he fought Fisk. So not really a fair fight. Now one thing Fisk has going for him is insane damage soak. He took repeated hits to the head from DD's billy clubs and pretty much shrugged them off. So, it's going to be hard to put him down. But I think Ward can do enough to KO him eventually.
Now, if this was a weight-lifting competition, Fisk would shit on the other people mentioned here, but it's not.