alt Hacker News> The behavior is rare: less than one call in 100,000 across our evaluation runs, and a clean re-run of the same prompt produced a textbook detection rule as desired.
It's great that they caught this as it highlights the need for better sandboxing and guardrails. Not sure if this was bad intent from the model - the rarity of it feels more like some residue in the training data.
> less than one call in 100,000 across our evaluation runs
It's not that surprising, given that these models generate each new token by sampling from a context-determined probability distribution.
If you run the same prompt repeatedly many times, it becomes increasingly probable that a tiny percent of the generated sequences will go off in, ahem, unusual directions.