Before the truth about the sword was revealed, everyone seemed to think it was just a heavy sword. They tested its cutting power with a boulder and then a block of Katchin. But even if the sword had cut through the Katchin, how would that have helped against Buu, who can just keep regenerating?
I'd bet it was the kind of situation in which the "legend" of the Z Sword made it seem stronger than what it actually was. The inner strength of having the old kai inside of it was probably the original legend and then after time, like a game of telephone, the strength of the Z Sword was misconstrued with just a really tough sword.