Hallucinogenic experiences can be really specific to particular hallucinogens.
There are some hallucinogens that only produce auditory hallucinations. We don't really have a reason to think those are real sounds.
There are others that will not make you visually hallucinate in the regular sense, but will mess with how your brain interprets what you see - scrambling your internal visual object classifiers, so to speak - so that distinct things sort of melt together in your mind.
Hallucinogens are wild. But I think they are the same as any other drug that has predictable effects, like alcohol or pot.
Maybe it throws us for a loop because subjective experience of hallucinogens are so bizarre and varied and often intense. They truly do weird things to us.