It is not generally legal to drive while wearing headphones. In some US states it is specifically banned, and in many others you will get pulled over for distracted driving. (The thinking is partly because it makes it more difficult to hear emergency vehicle sirens).
If your headphones are blocking sound, yeah it can be hazardous.