Depending on your definition of poison of course, but at least for LSD and magic mushrooms, all that's known is a lower bound for their biological toxicity. For LSD IIRC it's somewhere in the 30mg (!) range, for mushrooms it's more than can comfortably fit a stomach. I'm afraid I don't have sources handy though.

That said, the state of mind must be so wild that "is it poisonous in this specific/usual sense" is probably not the right thing to discuss.

> Many (perhaps most) drugs cause harm at the wrong dosage.

Not only drugs, almost everything is bad at the wrong dosage. Case in point: water.