The weak link in the Silver Samurai is the human inside. Steel (adamantium or whatever) is not a good shock absorber. So a full powered punch from the Destroyer may not dent the adamantium armor but it sure as well will give that old man inside some blunt force trauma. A couple of full powered shots from Destroyer should turn him into mush.
Plus, repeated eye blasts from the Destroyer will eventually heat up the adamantium suit, burning that man inside.