The binary file problem is real - audio files balloon Git repos fast. Git LFS helps but adds complexity. I've had better luck with a hybrid approach: Git for project files/metadata/stems, cloud storage with versioned folders for final mixes. Keeps the repo lean while maintaining the branching workflow benefits.
I've never really used it for anything serious, but doesn't Mercurial handle binary files better? I thought there was a more clever binary diff system for it.
Are there any music formats that allow, conceptually, for easy diffs?
If there are, it's not beyond reason to add something to git to make it work better.
If changing a single bit at the start of file changes the whole thing then it's really a failing of the file format. By which I probably mean the container format.
I’m sure if GitHub made LFS storage free, adoption for large projects would 100x, and LFS bugs would be fixed fast.
I'm gonna guess that the OP (and many others) might use Git purely for the source file. I don't know about other DAWs but Ableton which is my DAW of choice has .als (which is usually pretty small) which contains references to the WAV and other dependent files. Of course, this doesn't solve the (common) problem of losing all your audio files but it does maybe make this approach a bit more feasible...