There’s physical dependence and there’s psychological dependence. Most drugs can cause both, but hallucinogens in particular are usually thought to cause only psychological dependence. Whether that makes them less dangerous is debatable, but the fact is, they can still cause addiction if used carelessly.
Now to your main point... dopamine hits aren’t inherently good or bad. They can, however, also make things addictive, and drug abuse is indeed a good parallel here.
What do you think about pinball? Is it bad for us, should we sue?