I'm glad you said something, Evie. Sometimes we feel this pressure to be polite and not bring attention to these casual racist comments, but then the speaker likely thinks we endorse the idea.
I don't think it needs a big dramatic yelling match. That immediately triggers a defensive reaction in many people. Instead, I think your strategy is more effective. A casual but direct question when you're not in the spotlight with an audience puts that person on notice: I understood what you were insinuating but I don't agree with it.
I don't think we're going to change attitudes overnight, but it's a small improvement if we can get these folks to stop acting on their beliefs/speaking aloud. I subscribe to the belief that casual racism continues because it's being taught/modeled to the next generation. If we can get the adult racists to shut up and stop talking about it, maybe it won't be seen as "normal" by the next generation. We'll still have racists, but it won't be accepted as "that's what everyone thinks." It will just be "that's what those people think."