You should read a little about it and I think you would change your mind.

The adverse effects of psilocybin are well known and documented and we know for sure that it’s not physically dangerous.

For the trip part we mostly know how it work and how interact and it can be healing (or bad trip depending on the context).

Still you may have have digestive adverse effects, not because of poisoning but because the human gut cannot digest crude mushrooms.

But you will not be physically sick per se like you would be with a bacteria or a toxin, your immune system will not fight anything.