That sounds awkward.

The tech for isolating a speaker at conversational distances exists. You use half a dozen microphone transducers (minimum; Crappy microphone transducers are cheap and quality is expensive, so just use a bunch of them), and through a combination of phase and intensity they decode relative location, and amplify that phase expectation while suppressing everything that isn't phased like that. Sound is slow, and readily susceptible to real-time triangulation. The math/processing is much easier if the parallaxes are fixed (eg the microphones are arranged in a line array on the top band of a rigid pair of smart glasses), but with a little latency it's not prohibitive for a deformable array to solve for its own relative position as well.