If your "friends" were calling you or your other friends pedophiles (and you are not) then yes, absolutely, you should not be friends with them.