The white of Harlem

In Harlem there is predominantly black people or African-Americans as they are called here, and if not black the inhabitants will be Hispanic or Latinos as they are called here. If some one is not black or latino it will most probably be me.
Sometimes you meet a white person though, and very rarely you see an Asian.
One interesting thing I have noticed when encountering whites in Harlem is how they avoid eye-contact as best they can. Otherwise staring is not frowned upon here: there is plenty of people to both stare at and on you. Might also have to do with how I look but I am somewhat used to getting stared at. But when you meet someone white they really go out of their ways not to lock eyes with you by accident, in all cases but one gay man in the grocery store.
I have a theory why this is: I think they do not want to look at a strange white person and there by get the feeling of "us" versus them. They are afraid to be seen as racist. But this idea is a bit of a stretch, even though I have not found any other. So if anyone else has some input here I would be most interested in hearing it!