If the RPM/deb comes from a Linux distribution then there is a good chance there is a separate maintainer and the binary package is always built from the source code by the distro.

Also if the upstream developer goes malicious there is a good chance at least one of the distro maintainers will notice and both prevent the bad source code being built for the distro & notify others.