alt Hacker NewsAlcohol requires an astronomical intake compared to most drugs, and I'd say the indiscriminate action and lack of any particular affinity is what makes it a poison as opposed to a drug. It is also generally assumed that alcohol is not exactly in the same category as drugs. ("drugs or alcohol" etc.)
That's purely because it's usage is accepted nearly as a default, and drug in common usage is most often a pejorative term if it doesn't refer to a medicinal context.
Most people who use drug as a pejorative term also wouldn't use it to describe caffeine.