> less than one call in 100,000 across our evaluation runs

It's not that surprising, given that these models generate each new token by sampling from a context-determined probability distribution.

If you run the same prompt repeatedly many times, it becomes increasingly probable that a tiny percent of the generated sequences will go off in, ahem, unusual directions.