I've did a bunch of recreational drugs growing up, but mushrooms were the only one that I swore off forever. It just didn't sit well with me and would lead to a ton of anxiety both during and after taking them.
I think it's too easy for people to get caught up in the idea that these are miracle cures and forget that, just like with any drug, the effects will be different for different people. I'd love for it to be available for people who are seeing benefits, but I don't think there's any shame in people saying, "that doesn't work for me"