The extreme case does not imply a binary scenario ie that there are those that can those that cannot.
Rather, learning ability is a continuum. people have varying degrees of ability to learn mathematics. Couple this with environmental factors and society generates a huge variability in mathematical ability that crosses income levels and other demographics.
This view is rejected by many because it is against the push for equality.
You get a huge variability if you consider the absolute extreme outliers. Most people should be able to reach a level of competence where they can understand mathematical concepts abstractly and apply that same reasoning to other areas, and not feel a visceral rejection at the mere idea. I think that's a modest enough standard that a good portion of any given population should be able to reach, and yet education is failing at achieving that.