In the quaint olden days there used to be a concept that the appearance of impropriety was as bad or worse than actual impropriety for exactly this reason.