People are pretty expensive (they're literally worth a lifetime of free labor). 1 slave would've been like 3 complete years income for an average free white southerner to purchase, plus ongoing expenses obviously. So they basically end up in the hands of upper class people who have a steady need for lots of manual labor. Doesn't mean that everyone else around was not benefiting from the economic surplus, or was not supportive of the institution.