The things I advocate are just and good. That's why I advocate them.
That means that anything else anyone else advocates that is different is unjust and bad, because they advocate what is unjust and bad. At the very least, by hypothesis, since I advocate what is just and good, when they disagree, they reject what is just and good and instead advocate what is unjust and bad.
To be friends with such people means countenancing their endorsement of what is unjust and bad. Moral purity and respect for justice demands I not be friends with anyone with whom I disagree. What am I supposed to say, "Oh, sure, Bob rejects justice and advocates evil, but I guess I can look past that since we have so much fun playing flag football and since he tells good jokes?" This seems to trivialize justice.
You might say, oh, but you could be wrong about what is just and good. Of course, I could be. But when you say I might be wrong, what I hear is that you think maybe racism, callousness, unfairness, rights violations, and other injustices might be OK. I probably shouldn't talk to someone like you who treats justice as if it were mere opinion and subjective taste.
You might say but can't the other sides say the same thing? Well, they might say it, but who cares? They advocate what is evil and bad, and the fact that they think it's just and good isn't an excuse. It's part of the problem. That Hitler was sincere and thought he was doing the right thing made him worse, not better! Hell, I'd rather be friends with someone who did the wrong thing and knew it was wrong than someone who sincerely believes what is evil is good. The sincere Nazi is worse than the opportunist.
You might say that didn't this kind of attitude cause all sorts of problems in the past, with the wars of religion? Didn't the idea of tolerance arise in part to solve the issue of instability and conflict? Well, that's a silly argument, isn't it? If there were a God and the One Right Way to worship that God, then why would I want to be friends with people who are like, "Oh, I guess I'll reject the actual God in favor of some silly nonsense I made up while sitting on the John"? Of course, the issue here isn't intolerance, but the substance of their debates. Fighting over religion and related behaviors is silly, while actual justice--actual anti-racism, for instance--is a real thing. When I read the Scarlet Letter, I think, "Good methods but wrong issue."
You might say, "Ok, I get that some views are beyond the pale, but surely some disagreements are reasonable and you should feel free be friends with reasonable people." Response: So you're saying it's reasonable to advocate injustice and evil, as long as it's a sufficiently little bit? Precisely how much racism or endorsement of oppression, exploitation, and injustice should I tolerate from others? Do tell.