This gcc script that I created below is just as "deterministic" as an LLM. It produces the same result every time. Doesn't make it useful though.
echo '#!/usr/bin/env bash' > gcc
echo 'cat <<EOF' >> gcc
openssl rand -base64 100 >> gcc
echo 'EOF' >> gcc
chmod +x gcc
Also, how transformers work is not a spec of the LLM that anyone can use to learn how LLM produces code. It's no gcc source code.
You claimed they weren't deterministic, I have shown that they can be. I'm not sure what your point is.
And it is incorrect to base your analysis of future transformer performance on current transformer performance. There is a lot of ongoing research in this area and we have seen continual progress.