One theory is that "Super Kaioken" would burn through his body at an exponential rate due to the level of energy required to sustain Kaioken and Super Saiyan. Of course, when Goku used said technique in the filler, he was already dead and him being dead allowed him to later use the Super Saiyan 3 form for a longer duration than he was able to when he was alive.
Personally, I think Super Saiyan + Kaioken is more or less unneeded when Goku can use Super Saiyan 2 or 3 when he needs to.
He did super Kaioken when he was fighting in the other world, however it looks like it took him from SSY 2 back to normal form yet he became more powerful then SSJ 2 at that split second.
Super kaio-ken wasn't used in the manga because it's too phucking haxxed.
Just think:
When Goku used that standard/power-doubling super kaio-ken against Pikkon, his power level instantly became equal to that of his SSJ2 form(as a SSJ2 is 2x more powerful than a SSJ1.) So if he were able to use higher kaio-ken multipliers, then he'd attain power FAR beyond what he ever displayed in the manga.
For instance, a SSJ3 is, per canon, 6x more powerful than a SSJ1(as it is 4x more powerful than a SSJ2.) If Goku were to use a x20 kaio-ken as a SSJ1(which is the greatest kaio-ken multiplier he ever used in the manga), then he would become over 3x more powerful than a SSJ3. Just imagine if he were able to use high-multiplier kaio-kens as a SSJ2 or 3... He could have soloed nearly every villain in Z at the same time.
I agree that it would result in Goku being ridiculously overpowered. But I also think the logically reason in-universe as to why Goku wouldn't use it would be because of the strain it would cause on his body. Barring Full Powered Super Saiyan, every Super Saiyan form is constantly taking a toll energy wise on the body. Kaio-Ken also does considerable strain on the body. Merging the two would likely give an absurd boost in power but at likewise absurd physical cost, especially if merging Kaio-Ken with Super Saiyan would logically result in power levels beyond that of Super Saiyan 3...and we saw how that form takes a toll on Goku.
^ As a full-power SSJ, Goku was able to stay in his SSJ form without putting any extra strain on his body. He'd mastered this form so perfectly that maintaining it was as effortless as maintaining his base form-- he could even stay transformed whilst sleeping, iirc.
So logically, using a kaio-ken as a FP SSJ shouldn't have put any more strain on Goku's body then it does on his base form. If anything, his SSJ body should be able to handle much more strain, given that it is 50x stronger than his base form. Conversely, Goku didn't have anywhere near that level of mastery over his SSJ2 or 3 transformations, so your point would definitely hold true to those forms. /shrug
__________________
"I am tired of Earth. These people.
I am tired of being caught in the tangle of their lives."
I've said it before myself. They've soften Vegeta up and they've been doing this every since the series was first running. He's just not the same badass that first bursted onto the scene/panel when we first saw him. It's kind of a letdown.