I think that agents cannot take over people that are aware of the Matrix and the existence of them (the agents). It is like a thief trying to get into your house while you can see him through the door window. You will not allow him to enter (block the door, lock it etc.)