Daredevil wasn't Colin Farrel's first major part. He had major parts in Minority Report and The Recruit and was the star in Phone Booth & Hart's War, all before Daredevil was released.
As for him playing Gambit, I think he'd be alright. Might be a little to old to be opposite Anna Paquin and still keep the love between Gambit & Rogue, but it could work. He is only 6 years older.