Makes sense. If you enable the Debug menu (Shift+CMD+D), there is an option for "Always-On Microphone". Might be worth a try to remove that latency.