It can affect shell subprocess startup time as well, which, depending on your setup and the tools you use, might be worth optimising for.

I don't remember when I did it, but it looks like I must have gone through this at some point (maybe due to using GNU Make a lot? Or perhaps it was some other tool) - my zsh setup does a bunch of autocomplete setup only in the interactive case, and it seems to help a bit with startup time, at least on macOS:

    % for i in {1..5}; do /usr/bin/time zsh -i -c exit; done # zsh in interactive mode
            0.05 real         0.03 user         0.02 sys
            0.02 real         0.01 user         0.01 sys
            0.02 real         0.01 user         0.00 sys
            0.02 real         0.01 user         0.00 sys
            0.02 real         0.01 user         0.00 sys
    % for i in {1..5}; do /usr/bin/time zsh -c exit; done # zsh in non-interactive mode
            0.01 real         0.00 user         0.01 sys
            0.01 real         0.00 user         0.00 sys
            0.00 real         0.00 user         0.00 sys
            0.00 real         0.00 user         0.00 sys
            0.00 real         0.00 user         0.00 sys

For the interactive case, I don't really mind (within limits - the worst case, on macOS after a reboot, takes several seconds, and that's tedious). I also start new interactive terminal sessions fairly rarely.