Dooku and Mace are on Revan's level although a bit below and is far more powerful than Novel!Meetra and Scourge. That trio were able to seriously threaten Vitiate, this team will win.
Revan has some of the best defenses of any fighter though. His tutaminis is arguably the best, his mental defenses are immense and he had special prep against Vitiate and his TK is better than Mace or Dooku's.
Dooku couldn't replicate Revan's fight. Mace could repel Vitiate's lightning through Vapaad probably, but not Vitiates other attacks....
Here's the thing, let's say this situation unfurls the same way it does in the novel.
Mace bats his lightning aside and puts Vitiate on his ass. He force storms and Mace gets overloaded (unlikely...but still). Vitiate moves to kill Mace, and Maul or Dooku throw a lightsaber. The difference between them and Meetra is that they'd easily sacrifice Mace to kill Vitiate.
Dooku and Mace alone could win against Vitiate imo, Maul in the mix makes this a solid victory against him, if not a stomp.
You forgot the first exchange, where the telepathy comes into play. I'd say no-one here has the chops to resist. If it unfurled like it did in the novel, it'd end up Vitiate and Mace vs Dooku and Maul.
The only thing that gave Revan a chance to harm Vitiate was the unexpected resistance Revan showed against Vitiate's TP which was possible only because he was prepared for it. If Vitiate chooses the engage them the same way he did to Revan, he will break at least one of them and stall the others with the same attack as they will still be assaulted by his TP as well and it should stall them enough for Vitiate to turn one of them into his slave. After that point, they don't stand a chance.
__________________ There's a man goin' 'round takin' names.
An' he decides who to free and who to blame.
Everybody won't be treated all the same.
There'll be a golden ladder reaching down.
When the man comes around.