Here is my theory:
My theory is that Slytherin's family branched out into two directions during the war against the last dark lord ( I forgot how to spell his name... L
L) anyway, I think that one side stayed on the dark side and the other went to the light side and had married a muggle. Now there child became a squib and supposedly so did the grandchildren, that is until Lily finds out about her powers, making it that Harry is the heir of Slytherin through his mother's side of the family.
Now, with the whole thing about Voldemort going after Harry and his father and not his mother is never truly stated in the series. It is just said that Lily needn't be killed, this could be that Voldemort didn't want to kill a heiress of his forfather and wanted her to join him and his deatheaters after her family was killed, yet he could just as easily kill her too.
So as I see it James was just the bother that Voldemort wanted to get rid of.
Finally, about the sword and how Harry pulled it out of the sorting hat. Dumbledore said that only a true Gryffindor could pull the sword out of the hat, yet that doesn't mean that Harry is the heir to Gryffindor, I see it in a different view. I see it as that it proves that though Harry may have Slytherin blood running through him, he is a true Gryffindor at heart and different then Voldemort. I imagine he will find this out toward the beginning of the seventh book or towards the end, either making him be in denial through the whole book and at the end see just how truely different he is from Voldemort or making Harry be hopeful through the book and at the end in denial... Either way, Ginny, will snap Harry out of his denial phase!