I'm not saying you're wrong on the limitations of these studies, but I also don't think it's unreasonable for people to believe in the power of these drugs.
For many people (myself included) taking psychedelics was an immediately life altering experience. I don't need any more scientific validation that psychedelics helped me than I would need to prove that touching a hot stove burned me. It's an immediate and unmistakable effect that is wholly different from any other experience I've had.
Now, proving that these drugs would be beneficial at a population level may be an unanswered scientific question. But, to quote you: "trying to prove psilocybin works for psychiatric illness" - well, that's been done. The people who have been helped have all the proof they need. I think the issues you've raised are more properly targeted at the question "should we recommend psilocybin treatment to depression patients more broadly" which is in fact a much higher bar than "do they work?"
I can respect this view 100%. And your rephrasing, "Should we recommend psilocybin treatment to depression patients more broadly," is in perfect sync with my thinking, and the right question to ask.
I would only add that it's extremely hard to know if something actually works. There are seriously some medications that people claim don't work, but objectively they are healthier, happier, more social, more active, and "better" on almost every objective measure. An example of this is when you really have a good match with venlafaxine for co-morbid panic disorder and depression.
Then there are drugs everybody claims are good for them, but they almost never are, especially in the long run. Basically, all of the benzo sleeping aids (in the context of chronic use).
So I can't and won't say you are not right. But I hope you permit me my goal of finding "objective" markers for saying "something works." Because, to quote Murderbot, "Humans are idiots!" (me included). And self-report is a notoriously unreliable measure.