All the package managers that provide shell wrappers kinda tend to be bad at this, unless they use their own command to wrap over project specifications, like uv.
These days, I've been personally relying more on direnv to automatically activate certain shell configurations, and then nix to manage binary dependencies like node or go or php.
In my experience direnv is also a source of slowness. How fast is it for you?
mise isn't, and has the advantage that you don't need to build your own lightsaber with direnv and nix.