Before the truth about the sword was revealed, everyone seemed to think it was just a heavy sword. They tested its cutting power with a boulder and then a block of Katchin. But even if the sword had cut through the Katchin, how would that have helped against Buu, who can just keep regenerating?
I'd bet it was the kind of situation in which the "legend" of the Z Sword made it seem stronger than what it actually was. The inner strength of having the old kai inside of it was probably the original legend and then after time, like a game of telephone, the strength of the Z Sword was misconstrued with just a really tough sword.
The z sword could only be welded by those more powerful than the kais and held within it the power to maximize your potential. They just missed the part about that power being an old hermaphrodite