With zsh i set up nvm to lazy load, so I don't pay for it when I don't use it (I'm a C++ and Rust dev, but I occasionally need to run js stuff from other team members).

I can strongly recommend lazy loading in zsh in general, I use it for pyenv too (which is also slow to load, but I write Python maybe every other week or so only).

The way to do this is to use the autoload functionality in zsh and have the autoloaded script replace itself with the real shell init code for the tool in question.