The Governor plays a vital role in a secondary-type story while the group is taking shelter at the Prison... So if they do that part of the show right (and by "right" I mean do it like the comics), it *shouldn't* seem rushed. After all, the producers of this show don't really seem to "rush" anything.
On another note: I am thoroughly awaiting the arrival of Michonne. The group desperately needs another badass aside from Daryl. Hopefully the rumors are true and she will debut during the season 2 finale. *crosses fingers*