here's the thing, for all of batman's skill he had enormous trouble taking down a mere ninja in the last part of the knightfall series. yeah he was still recovering and yeah the ninja was big...but there was really nothing special about him. Batman as shown on more than a couple occasions that oponents he isn't prepared for can hand him his ass in h2h combat (i.e.the reaper). All taskmaster has to do is last a few moments with bats, and he'll already be picking up batman's fighting style. batman's got tons of tricks sure....but so does tasky. eventually batman will run out of techniques to use, and taskmaster will have his ass owned.
comparing deadpool and batman at hand to hand combat is like comparing apples and oranges. both still fruit but both are verydifferent from eachother....Deadpool caught tasky by surprise because he was so radically unpredictable. Batman on the other hand is a proven tactician.....every move is precise, every technique and every counter calculated.....he doesn't have the same randomimity (as if that was a real word) going for him that deadpool does.
in anycase batman and cap are practicallyequal in combat, taskmaster knows all about cap and he'll be able to last long enough to know all about batman as well. after that, it's goodnight bruce wayne. if an old guy in a leather suite could do it, taskmaster is gonna do it better...with an attitude.