The way I usually solve this is by using git submodules.
AFAIK the issue with using submodules is you still need the rights to pull the other source repo. However, you can use submodules or LFS to pull a specific build artifact from a build artifact repo or source instead of the source repo, which provides a neat way to manage the dep without fattening the main repo and allows the source repo to be kept separate and high security. I'd certainly do this before changing RCS/VCS solutions. That said, reverse engineering has become relatively trivial in the AI age so the practical utility of providing built rather than source elements is dropping.
Oh man, I've been laughing at this for 37 minutes straight now.