A swedish blogger put it pretty nicely a few weeks ago regarding moderating his own comment field: It's not about filtering out things you don't want to hear, it's about filtering in a usable and valuable conversation, i.e. removing things that do not contribute to that conversation, such as abuse, insults, and all kinds of disrespectful and domineering discourse.

Also, people who do not want to participate in a conversation, but rather just spout crap, or get cheers (and jeers), and specifically, not actually read or listen to others (as nazis and that ilk are prone to doing), should indeed get banned.

In the !fediverse, It's really about what you want to host on your own computer. If you want to give nazis a platform, feel free, but when/if I run my own server, I'll feel just as free to not federate with instances hosting nazis (or spam for that matter), since those users generally do not want conversation.