I do that. This was never a problem, as the ANC ones I used don't cancel every sound the same way.

For example, I can go into datacenter and it will cancel all the datacenter noise(aside for when air blows directly into mic, it overdrives it) but I can still hear what other person is saying.

Also I used them to generally listen to podcast so there was no wall of music to go thru, so sirens and such were easily discernable

Do you also think drivers with windows blocking sounds and their stereo blasting are mad?