In practice, building from source is not going to fix the problem. Nobody reads the source code of projects they download and compile themselves, certainly not for larger projects. It also takes a long time to compile larger projects. So, realistically, these rarely happen.

Of course, the one advantage of having source is that it is easier to run things like SAST tools against source, but how many people do that in practice? How integrated is that with package systems? And when package maintainers might provide hashes of what they ostensibly checked, you still need trust.

So we need a combination of static analysis tools that are integrated properly to produce trusted binaries, and you need earned trust and authority. Hyperindividualist self-reliance is, at the very minimum, impractical. And with authority, we know whose job it is to care for the quality of software and therefore whom to hang.