I am frustrated and disheartened that this is your reality and you have to pass this lesson on to your kids. It immediately made me think of "Whistling Vivaldi" by Dr. Claude Steele. He writes about stereotype threat in his book and the title comes from a story his friend, then a young Black man, shared with him. The friend found that whenever he was walking down the street, he tried to reassure scared white people that he was not a threat, he's an educated Black man, by whistling Vivaldi.
It reminds me of your Polo shirt as a visible signal but takes it a step further since Vivaldi is such a stereotypical upper-class music of choice.