I built something very similar and comparble to this with wakeword detection on my raaberry pi.

Groq 8b instant is the fastest llm from my test. I used smallest ai for tts as it has the smallest TTFT

My rasberry pi stack: porcupine for wakeword detection + elevenlabs for STT + groq scout as it supports home automation better + smallest.ai for 70ms ttfb

Call stack: twilio + groq whisper for STT + groq 8b instant + smallest.ai for tts

Alexa skill stack: wrote a alexa skill to contact my stack running on a VPS server