If most people want to live near others like themselves regardless of their ethnic backgrounds, that already implies that non-white peple wants to live near non-whites like them rather than whites. Thus, there is no reason non-whites think whites racist only because they want to live near people with same ethnic backgrounds.
However, if any ethinic group tries to push out the other ethinic group by harrassing and/or thretening those people, they are racist. There is huge difference between wanting to live near people like ourselves Vs trying to kick out people different from us. For example, nobody calls white flght from Cupertino racism.