This is a somewhat recent new "feature" to force group calls, even if they're accidental. It's not what most people I know want, and there is no way to disable it for a group, just as there is no way to disable audio messages anywhere. WhatsApp is made to the lowest common denominator, UX is secondary to market share.