Also, I don't think it's ever been confirmed that a stronger soul makes the Destroyer stronger outside of Odin infusing the Asgardians power into the Destroyer. It makes sense. But it's never been confirmed in my recollection. The one time the Destroyer actually killed Thor, a simple human was inside.
You can make the argument and human soul does not make the destroyer weaker. But stronger soul does seem to make it stronger. Ya thor has been owned by human spirit at least twice.
when hela and her goons had the destroyer (please log in to view the image)
Here Thor claims that the destroyer is stronger than before.
There has been more than one instance of thor getting owned by Human spirit destroyer.
I find this interesting that the destroyer actually becomes stronger each time it is animated. <doesn't seem to be true but interesting>