The vast majority of the human population is lactose intolerant, both historically and today. Genetically intolerant populations in South and Central Asia have microbiotic help with their dairy-heavy diets, but for people who didn't spend thousands of years developing a culture around it, dairy is just a quick road to an upset stomach and/or food poisoning.