The watermark lives in the entropy of sampled outputs. Typical entropy of sampled English text is about 1 bit/token, meaning that a 500-token response from a given model might have 2^500 potential outputs of roughly equal probability. The watermark restricts the sampler to some subset of these - say, 2^400 of them, so chance of accidentally generating a watermarked output is astronomically small (2^-100). As long as the restriction doesn't condition on the content of the samples themselves, the watermark is "non-distortionary": the outputs are all still samples from the model's original distribution, and so will satisfy all the same statistical properties, including things like expected performance on any benchmark or eval you can construct.

In cases where the output has low entropy - eg, you've asked a model to repeat some input text verbatim, or to answer a question that has exactly one correct answer - there will be no randomness for the watermark to hide in, so the output will effectively not be watermarked. Code lives somewhere in the middle: it generally has less entropy-per-token than prose, so would need more tokens to reach a given level of detectability.

There are lots of ways to restrict output samples. The simplest conceptually would be to just use a restricted pool of PRNG seeds, but in practice there are more sophisticated constructions to try to build in robustness to minor edits, allow detectability without needing the original weights and prompt, etc. Google's SynthID paper (https://www.nature.com/articles/s41586-024-08025-4) is a good starting point if you want to understand a recent production-ready method (or you can just ask an LLM to explain it to you).

This is a good explainer: https://magazine.sebastianraschka.com/p/claude-watermarking

It does change the output, they never said it did not. They said it would not _noticeably_ affect performance.