me having played the game would not chance my argument. I know the primary facts.
This is false.
It would, because you don't.
Ganon does keep tabs on Link, and he does do what he can to stop him from getting what he needs to put him away, as well. (You think temples randomly get full of monsters and bosses by accident?) If you'd play the games you'd know that each time he's lost has been because of circumstances that can hardly be called fair.
OoT: Link had already drawn the master sword, which sealed him away in the temple of light, unaccessable to Ganon for seven years. When Link returned the triforce had been split, and Ganon was using Link to bring Zelda out of hiding.
WW: He needed Link to pull the sword from it's pedestal to break the seal on his power which was placed on it in OoT.
TP: Alternate timeline, had no idea about the events of OoT other than what happened in the child timeline resulting in his execution by the sages, which was botched, and resulted in him being sent to the realm of Twilight. Zant did have the chance to beat Link, but screwed up by not killing him. Zant failed, not Ganon. Ganon wasn't present until the end of the game.
aLttP: Ganon is sealed away in the sacred realm, and cannot deal with Link directly.
Ganon given ten years of prep-time and the magic mirror Miria mentioned would have none of these problems. (No Link with unfair advantages.)