I imagine that makes having the settings be specific to each source even worse. How else are you going to adjust them for navigation instructions?
My car has something like that, but thankfully I have only needed to adjust volume, which can be done from the steering wheel…