It's not the 'specific determinism' that is the problem aka 'slightly different answers' - it's the 'general nondeterminism' that is the problem.

Aka the 'hallucinating' is the problem, not the slight variation in possible outcomes of a given prompt.