Isn't the magic however somewhat weaker? Like the same trait when used with unspoken spells. Like could the killing curse be performed with a wand and still kill someone or is the use of the wand purely for show "Ha I can kill you and look cool doing it" etc. Also Gandalf would not need to use magic to break a wand if he could get close enough he has a sword, a rather powerful sword and since I last knew hitting wooden objects very hard with big metal objects usually results in some form of damage.
Also I would at most say Voldermort was of equal power to Saurman and Gandalf since as Curumo and Olorin respectively they did kinda have a hand in making well the whole world, they were immortal spirits and were of a form of "angelic order". Whilst Saurman still fell into shadow he did not loose his power for Gandalf to shatter his staff was a show of great power. It is speculated that Gandalf as the White was the second most powerful being in existence save the one that sat in Barad-dûr.
I personally think Gandalf has this one in summary
1. He is an immortal spirit simply killing his flesh could not be seen as a victory
2. He has a very high resistance to physical harm, to fight the Balrog for so long he would have to.
3. Gandalf may use with competence both sword and magic to disarm his enemy or slay them.
4. Gandalf is somewhat more versatile thinking on his feet he one even lit pine cones on fire and used them as flaming projectiles.
5. If it came down to both of them killing the other, Gandalf would not truly be dead his spirt would live on.