It isn't hard to program a gpt. You can do it in a weekend with a few hundred lines of python. The code is pretty simple. The math is not particularly high level.

The complexity and scale with LLMs come from the amount of training data used, not some kind of black magic in the programming.