If we go by "X" the movie, which afaik was fairly accurate, his trip to Mecca, where he worshiped with people of all races, was hugely transformative in his opinion of whites and racial integration.
Also, there is a line where a white girl asks him what she, as a non-racist white person can do to help his cause, to which X says "nothing", sort of indicates that he was only preaching against white establishment and not individual white people.
The problem is, X was alive in a time where all the establishment was essentially "white". To fight the battle he was fighting, some aknowledgement of this would have been necessary, and in a way, that does make him "anti-white", but only in so far as whites controlled the systems that legitimately work to oppress black people. We could probably call this "racism" in a philosophical sense, but I think there is very little evidence that this extended to X's opinion of white people in general.
There are good interviews of him on youtube, where he seems to have little problem discussing with white people, though his confrontational nature does come out.