Fair questions and criticisms.

I would prefer to have colleges [elite or otherwise] be generally representative of the population by race, religion, socio-economic background, etc.

The problem, as I understand it, is "best qualified" is massively impacted by things like race, parental income, and other factors outside the control of college applicants. Society is far from a meritocracy.