Your guards, sure, and any settlements that would have seen their approach. But Ogros has a small army out there ready to destroy Firak's entire setup.
In game terms, if you force Firak to bring all his men to defend you, you will permanently lose access to your Epireans because Ogros will kill them.
You were talking about bringing in Firak's forces! He needs them- including the men you are currently with- to defend against Ogros. If he leaves his lands defenceless, Ogros will destroy the lot.
He's also very concerned by your idea that they are going to attack his settlements. He appreciates your magic powers that have forewarned you, but he has absolutely no idea why they would do that- why fight more people than they need to? But if it is going to happen, he wants to evacuate his people.
If you are talking about his people, he objects to you just bringing the women and children up there to be killed. If you are talking about his warriors, he will say again- he needs them to defend against Ogros! Just because you've turned around, it doesn't mean Ogros is going to go away. Ogros may not attack your at your base, but he'll still go for Firak.
He wants to organise as many warriors as he can for defence, and if you are certain the invaders are going to attack his settlements- for whatever reason- he wants to move everyone out.
He wants to move equipment out too, unless you want Ogros stealing all the technology you've given Firak over time.
Those are legitimate concerns. We don't want Ogros gathering the technology we've given Firak and we don't want to lose access to our Epireans. Galder is not going to make his people stick around just to be murdered; it's a waste of resources.
Is he willing to loan us any men for the defense of our base?