How does your "anyone knows" control for false negatives? i.e. how can you be certain that the people you met who aren't "anxious and stupid" aren't also weed addicts?

What's wrong with people choosing to do drugs if it has no major adverse effects on other people?