That's simply not true.
Until spec files can be worked with so efficiently that we can just regenerate the binaries from them for every change, what you said makes little sense.
You're going to need an intermediate portable form of storing the program. It so happens that all the tools and AI are optimized to work with source code.
Exactly, at what point do spec files become so precise that they are just another form of source code?