Yeah, that fight is the only reason anyone backing Cap has a leg to stand on, and it was so laden with context from the angst-ridden Pete that we can't really draw a conclusion from it. Why people can't understand that, in comics, emotions play an absurdly significant role in fights, is beyond me. If I had a nickel for every time Pete shuffles his feet around awkwardly, gets beat up, then takes the kid gloves off the 2nd time around and absolutely destroys a supposedly equal or superior opponent...I'd have at least a few dozen nickels.
And if their fight corroborated the other evidence we had, it wouldn't be a big deal. But it contradicts literally everything else we know about the characters.