I am developing my own programming language, but I have no specification written for it. When people tell me that I need a specification, I reply that I already have one - the source code of the language compiler.
The source code does what it does, including bugs.
So unless you want bugs to be your specification, you actually need to specify what you want.
A programming language is not the compiler. A programming language is, in fact, not software.
Maybe this is your point, but the source code of any non-toy compiler is not a usable specification for the language it compiles.
If you want a specification from source code, you need to reverse engineer it. Although that’s a bit easier now, with LLMs.
You are not wrong. But, they are not wrong either.
I feel like if you’re designing a language, the activity of producing the spec, which involves the grammar etc., would allow you to design unencumbered by whether your design is easy to implement. Or whether it’s a good fit for the language you are implementing the compiler with.
The OP also correctly identifies that thoughtful design takes a back seat in favor of action when we start writing the code.