I agree it can be weird to realize you may be a person's only friend of color unless you live in a place that is 99% white. In those places, I'm sure there are white Americans who have never met an Asian American, Native American, Latinx, or Black person before so all they know about us is from TV.
I think I'm especially sensitive to it as I don't want to feel the pressure of representing my entire race. I also don't like it when people say, "I'm not racist. I have a [Black/Asian American/Latina/Native American] friend."