This is not an issue for me due to my workflow.

I have a script for each of my projects that I run when I open a new terminal window (Alacritty). The scripts set up tmux with 3-8 terminals, each terminal launches a components, utility or just sits in a folder from which I later run commands.

Having said that, I use only a few zsh plugins, and have a theme configured to not run commands that add extra latency.