The term "Slavery" used in this context doesn't refer only to the ownership of Africans, it's also a catch all for the subsequent segregation/Jim Crow laws put in place by this country up until the 1960s. Some people (myself not included as I try to deal with each person as an individual, not the sum of their descendants transgressions as, for example, I wouldn't expect the actions of the Moors, Cushites, Kemites, or Zulus held against me) hold all White people accountable due to the current day power structure being in place via the subjugation of non-whites here in America. While historically there have always been impoverished whites in America they were still treated with the benefit of citizenship and humanity before My People, even accounting for the horrible treatment of the Irish, Italians, and other groups here in America. We had been the actual bottom of the metaphorical totem pole here for centuries.
By contrast, though, you also have to take into account that many of the people who hold this position also don't have the benefit of historical perspective and either disregard or are ignorant of the treatment of European "minorities" in this county, or the actions of European and American Abolitionists (like William Wilberforce, for example).
Basically, pun not intended, history isn't as black and white as contemporary schools of thought would have you believe.