You'd have to be brutal about culling, uninviting and removing anyone who doesn't look like a good fit.

Or have a two-stage process: run very public, very open events that anyone can sign up to an attend. And then invite specific people that you meet at those events that look like a good fit for your community to your private, community-only event.