That's purely because it's usage is accepted nearly as a default, and drug in common usage is most often a pejorative term if it doesn't refer to a medicinal context.

Most people who use drug as a pejorative term also wouldn't use it to describe caffeine.