The issue with binaries is when they are large, change often, is often a duplication of information already present, and that you can't diff them. A program is neither large, it doesn't change often, is not duplicate information and you don't want to diff them. IMHO this idea is fine, this is not added the build artifact in every commit type of situation.