In my opinion, it's relevant to Card's credibility. If he shows poor judgement in one area, why would I want to listen to his opinion on something else, even something which is considered to be in his wheelhouse? Poor judgement is poor judgement.
Everyone is very, very wrong about something. By your logic we should ignore everyone about everything.
Everyone shows poor judgement in one area.
It sucks that he has those ideas, but he's a good writer.