It actually can since its just symmetric encryption. Any key holder could decrypt the payload. In fact, the channel could simply be the shared secret.

Let's say we have 10 people in a call, 5 share a key and the other 5 share a different key. Without the shared key audio simply will not decrypt. You could have two private channels with one host.