No, that's not racist to observe that you live in a predominantly white town. I know there are many areas of the U.S. where you see very few people of color. I think that's why it can be hard to break down stereotypes. For example, if you've never met an Asian American in person, you may assume we all love math/science, we only use chopsticks to eat, or some other stereotype promoted in the mass media.
Then someone from a town like that might meet one of us and realize, "Whoa, this Asian American loves painting, writing, and singing and inhales pizza and burgers. She's just like my sister!" That's where the artificial "barriers" of foreignness fall away and we realize we're more similar than one might initially think.
Thanks for reading!