Thats not enough. This problem pops up everywhere people do CVE analysis. If a tertiary part of some library is hit by some arbitrary CVE, why should that affect the core functionality people actually use? If I use ChaCha20, and that's bundled in a library with md5.do I care that their md5 library has a vulnerability? No, not really.
If you keep driving the "seperate the packages" angle you end up with is-odd, and that not a nice play to be either.
At least for native binaries it's not too hard to write a scanner that extracts symbols from the binaries and checks them against the symbol names of the vulnerable code. If the code is properly dead and got pruned by the linker, it cannot cause any problems.
Even more than that, a lot of vulnerabilities require untrusted input to exploit, but the library or executable is usually only used with trusted input. For example, suppose the chacha20 function had a vulnerability that was exploitable with a malicious secret key, but you only call it with your own, definitely not malicious, key. The vulnerability doesn't impact you, but your security policy might still require you to update. And no amount of splitting the package will help with that.