From what I recall, caffeine is mostly a concern for folks with pre-existing cardiovascular conditions.
Most adults can build tolerance, and I believe some studies are showing potential links between caffeine consumption and positive outcomes.
Of course, things get a little weirder with higher doses, and I am a bit concerned about new methods like pre-workout powder.