I have the same problem, but I always assumed it was Apple's fault. I don't know why the HAs/Airpods have the final say.

I don't think there's a way around it on the iphone, but I was able to cobble a fix for my macbook at least. It uses Shortery to run a Shortcut whenever my HA connects. The Shortcut runs a shell script that uses https://github.com/deweller/switchaudio-osx/ to determine the built-in mic and switch back to it immediately:

BUILTIN_MIC_ID=$(switch-audio --list-input | jq 'map(select(.name == "MacBook Pro Microphone")) | .[0].id') switch-audio --set-input="$BUILTIN_MIC_ID"