the "latent space containing all voices" may give you the ability to parametrize voices and make an infinite number of unique voices. BUT... people have a limited ability to distinguish points in that space.

in perceptual psychology/psychophysics, there's the concept of the "just-noticeable difference" (JND) which is the smallest change to a stimulus you can make that is reliable detectable.

normally the JND is measured on physical properties like brightness, pitch, etc but there's no reason it couldn't be applied to a more abstract latent space. two points in a particular latent space may be mathematically unique, but if they're indistinguishable to humans we shouldn't treat them as distinct voices