Its also about language and colloquialisms. If you grow up poor, but go to a school with many wealthy children, you will learn to speak the "language" of the wealthy, which is, in my opinion, what mostly explains the "having potential" that the ruling class will see in your as you enter adulthood.