Well, the Star Destroyers and military in general were basically tasked of keeping the Empire together. Sure, the Rebels were a threat, but the Emperor still couldn't just strip away his military from the planets.
Strangely, if you look at the history of the ancient old republic, they resemble more of a combo of the GE, the Fel Empire and the Imperial Knights than the PT- Era Republic and Jedi Order....
The 3million Clones could just be a rapid response force that gets shuttled everywhere while while non-clones due the majority of the fighting. Although a better explanation is that 3 million units meant 3 million regiments or divisions.
Also, remember, the Rebel Fleet wasn't actually that large, just mobile.