Do you use it on a desktop? Mac by any chance? What's your setup?
I've been looking to find a simple and fast dictation app for English but almost everything I've tried (from Handy to many apps, eg some with Whisper in their names, after the model I assume) just don't work well. Apple's offering is worse than those though. I even tried with local enhancement models.
You should give VoiceInk[1] a go. It works great and has decent latency.
https://github.com/Beingpax/VoiceInk