It's hard to tell who would win because techincally no one has seen the Wanderer fight anyone his own size. But if we go by the generic "<character one> has beaten bigger, stronger enemies than <character two>" rule, than Wanderer would totally beat Link.
You can talk about Ganondorf or whoever all you want, but Wanderer has kicked the crap out of sixteen ridiculously huge creatures hundreds of times his size (and he took them all out a second time, in rapid sucession if you count Time Attack mode). Link has fought some big foes, but I'm pretty sure Link hasn't fought enemies THAT big before. Even though Link's enemies were probably a lot smarter than the animalistic Colossi, I still think size matters as far as Link and Wanderer's hero resumes go. So, I say Wanderer if you go by the enemies he has slain. But Wanderer has technically never fought someone his own size, at least as far as we know, and I think Link would beat him since Wanderer seems a little clumsy compared to Link. But I still like Wanderer more.
Going by Enemies Beaten - Wanderer
Going by Skill - Link, probably
I think it's also fair to mention that there isn't any kind of approximate piece of canonical info stating how skilled Wanderer is with the sword. But yeah, I still think Link is more skilled with the sword.