Yes, this isn’t a logical error at all. If you don’t have taste in one area — actually, it’s even worse, you’re not even aware of your own lack of taste — why would I trust your taste in another area?
Why would lack of taste in graphic design be even remotely related to lack of taste in food preparation?
It's like arguing you wouldn't trust a lawyer with a medical negligence case if they can't suture a wound.
Or you wouldn't trust a graphic designer with a restaurant logo if they can't make good scrambled eggs.
The best Mexican places I've been to in CA had decor reminiscent of a big-city bus station, despite how good the food was.