Parents can give their kids alcohol and cigarettes, that's fully legal.
Driving involves danger to other. We restrict adults from doing things that are a danger to others, it has nothing to do with being "vulnerable."