Smith may not be taking over everyone

They venture out, just don't stay out. They hang around the mero.

but that is still different from them "staying in the 'exile' section of the matrix", isnt it?

Agent Johnson: She means nothing
Agent Thompson: Find the exile

Agent: You are no longer necessary
Keymaker: We do only what we're meant to do
Agent: Then you are meant for one more thing. Deletion

The oracle explained that programs that do not choose deletion may hide out there. The mero choose to keep the older versions of the agents. They were too hard to kill. Why would the matrix get rid of agents too hard to kill. Cause the previous versions of neo could not defeat them. Hence the suprise when neo stopped the bullets. He is greater than previous versions.

nah, in my opinion you're wrong
lets just say that the 4th matrix was set 200 years ago, when people BELIEVED in werewolves/vampires. Although weird, people regarded these agents as myths, and they were hard to kil.
Now we go to the C20th, werewolves running around a city? Not quite.... A man dressed in a suit with shades, shouldnt case too many problems PLUS it is believable by *todays* (as in, 6th matrix) standards

perhaps a medieval matrix had dragons as agents, who knows??

yeah agents are pretty much just "elite" among regulars. and the matrix didnt recycle old agents, in the beginning of M2 neo fights a few agents and says, " ah, upgrades"