We're supposed to stand up to bullies, not follow them. We're supposed to stand up to discrimination. And we're sure as heck supposed to stand up clearly and unequivocally to Nazi sympathizers. How hard can that be, saying that Nazis are bad?
Remember, it's impossible to be "neutral" or "apolitical" when it comes to Nazis. Tolerating their behaviour and refusing to speak out in support of their victims only serves to side with the abusers. And capitulating to their disingenuous complaints and demands is nothing but cowardly complicity. So when you cave in to them, you have to ask yourself, "Was it really worth it, to cut off your nose to spite your face? Did you get what you needed, or merely what you thought you wanted?"