That's why I switched to Mise (https://mise.jdx.dev). Having new terminal sessions taking up hundreds of milliseconds because of nvm isn't the end of the world, but it's annoying when you're in the middle of something.

Mise reads in your .nvmrc files so you don't have to configure anything new.