Or, maybe, just not use LLMs?

LLM is just one model used in A.I. It's not a panacea.

For generating deterministic output, probably a combination of Neural Networks and Genetic Programming will be better. And probably also much more efficient, energy-wise.