alt Hacker NewsAnthropic's 'Watermark' Text Adulteration in Claude Is a Perversion of Writing
Comments
Crazy how a smart person like this fails to understand the gumbel softmax technique. It does not affect writing quality at all, provably. The very fact that there is generally no "best next token" with 100% certainty is precisely why the trick works (you cannot watermark a response to "respond with the To be or not to be soliloquy from the first folio Hamlet", for precisely this reason).
>I want any LLM I use to choose the very best, most precise words at every single decision point.
Does the author think he is currently getting T=0 output from Claude? Is he under the impression that T=0 produces the "best" writing?
This entire article just seems so detached from the basics of how LLMs work.
Translation: No one can ever again use Claude for proofreading their own prose unless they’re willing to risk that the whole thing might be flagged as having been generated by Claude.
I think that was intended, yes.
LLMs are already perversions of writing, so what else is new. Oh no, the over-long circumlocution generated by three autocorrects in a trenchcoat might be slightly longer because of this and maybe people will start noticing the subtle rhythms of vaguely peculiar word choices as yet another cue that you are wasting their time with machine-generated wordslop, what a terrible fate. Your long rambling walls of machine-waffling might be 37.05% longer than they need to be instead of the mere 36.58% longer they are now.
This article feels slightly incoherent. You want high quality precise writing and to use an LLM to generate it? Feels like those are diametrically opposed
> One of my fundamental problem with this is that no two synonyms carry the exact same meaning. “He leaped at the chance” and “He jumped at the opportunity” are very similar sentences expressing the same general sentiment, but they are not the same. The exact words we choose when writing matter.
Then why are you using an LLM to write? They're not capable of understanding such nuance. They do pick randomly between two synonymous phrases, they do not use some super smart algorithm to pick the one that sounds the best.
This excuse doesn't hold any water at all - Occam's razor says the author is just super annoyed that his AI writing will be identifiable as AI writing.
This is not meant to be snarky, But almost any writing done by Claude is a perversion of writing.
I honestly can't stand the way Claude writes. This watermark change just makes it scarier.
> The provider must mandate in their terms-of-service that users not remove the watermarking.
So, you don't own the generated text, and can't use it freely then. What if I copy paste a section, or rewrite a section of text to my liking? What if I rewrite some lines of code that contains the mark?
Security theater, and vague enough to be used as a weapon against who the government wishes.
I hope it's left off for non-EU customers.
> I chose to depend on a private company to express my own thoughts and now I'm mad that I'm not in control of the output
Who could have seen this coming???
It seems fine. I use an LLM to argue with me prior to posting blog posts so that I don't post obvious incorrectness, but the UX element to it is that it constructs notes about various sections of the text and we talk about those. There's no way for the generated text to enter the blog unless I copy-paste it and I'm not going to do that because the entire point is for me to write it.
At the point that you're generating entire volumes of text from Claude you're not really trying to be a sophisticated writer. I don't see how it's going to hurt for it to choose random related words.
Some here are arguing that mechanisms used by LLM providers already derail the goal of "the very best, most precise words at every single decision point", therefore the author is misguided.
The author has expressed a preference. Assume that there is a sequence of tokens, such that it is considered the absolute best by the author. This particular method of watermarking makes it less likely to generate that sequence, by definition.
I feel their argument would have been clearer and stronger if they had spent more time exploring the alternatives, and whether these alternatives would be just as effective. It is trivially easy to remove invisible tokens.
Like it or not, there is a public good to being able to identify AI generated content, and a small degredation in quality is tolerable in my opinion.
I don't think anybody has to worry about this issue though. Manual writing, coding, and proof reading continues to be an option. Where AI output is nothing to be ashamed of, the tools are available. For everyone else, there will be LLM providers that ignore EU law.
Claude's writing was already easy to recognize. The fact that Anthropic complied without complaint makes me wonder if they already watermark their outputs and used the opportunity to create goodwill. Presumably they want to avoid training their new model on text generated by the previous model, so they have reasons to be able to recognize AI-generated text.
I dont see how there would be remotely enough entropy in most model outputs for this to be close to feasible with any kind of accuracy.
Either they false positve on pretty much everything ever written, or the chances of catching a true positive is so low as to be useless.
Basically Cinavia for text, and that often falls over and is easy to remove even when there is megabytes of data streaming over a long period of time rather than 2 or 3 bits per wall of text, let alone what most people use claude for, when there is a strict dictionary and other tight output constraints.
> I want any LLM I use to choose the very best, most precise words at every single decision point.
Try running an llm like qwen 3.8 27B in Q8 locally with an intentionally very low temperature setting, it will write like a caveman crossed with a robot. You may find that an extremely literal output does not look pleasant to read for humans.
I was initially surprised that Gruber was so invested in the "quality" of AI-generated text, which in my mind is an oxymoron. But really, Gruber's interest here is with the EU. This forms part of his ongoing attacks on the EU, all because they have been forcing Apple to align with regulations.
Watermarks are garbage because they may embed account id, IP address and deanonimize you. That's why we should be using open-weights LLM whenever possible.
There is no coherent position in which the watermarking is a perversion of writing but AI writing as a whole is not a worse one.
Watermarking will be one more nail in the coffin of proprietary models if the world is so fortunate.
Reminds me of printer tracking dots.
I'd encourage reading this paper, and literature on scaling laws in autoregressive models: https://arxiv.org/abs/2303.11156
Total variation distance has been measured to decrease as you scale a model, and that is the primary mechanism "watermarking" as discussed in the Anthropic announcement relies on. It becomes more difficult to reliably detect text as a fixed sample count without tweaking the distribution further. Either way, it's a minor problem that will be addressed over time, compared to the issue of who can detect this without guessing or developing their own sets: providers not releasing a way to detect any such watermarks without going through them makes this entire approach hostile to the public. The EU regulation on this subject is interesting, although again most certainly not the primary driver for these practices:
"1.1.2: Signatories will ensure that AI-generated or manipulated content is marked with an imperceptible watermark, with the exception of very short text. For free-form text longer than 200 tokens, watermarking still needs to be applied, even though it may have lower reliability compared to that of watermarking very long text"
A proper, effective and useful law would have required providers to regularly release datasets to run your own verification on any text released within a fixed interval of time, presumably once out of rotation. Instead, it only talks about exposing an user interface going through their own services:
"Signatories will ensure access to their detection solution through a user interface appropriate for the audience of end-users that may eventually be exposed to the content generated or manipulated by their AI system. [...] Any restriction to the access will be limited in time until more reliable and robust detection mechanisms have emerged and have been adopted as the state of the art for detection mechanisms for the watermarking of free-form text evolves."
Most interestingly, in line with the EU's mass-surveillance program, an alternative solution to watermarking where it may not be sufficient is also suggested, although only optional for now:
"Where appropriate and taking into account potential trade-offs related to privacy and security, as well as scalability challenges and costs, Signatories may implement as an optional supplementary measure fingerprinting or logging solutions for AI-generated or manipulated content which allow for checking whether content has been generated or manipulated by their AI system. For example, direct logging may be appropriate for text content, whereas fingerprinting approaches may be preferable for audio and visual content."
it serves to show just how little regard the people behind these generated-text fingerprinting schemes have for the actual craft of writing.
LLMs have never been the place I've thought to expect any commitment to the craft of writing, to be fair.
It's good to be the King.
And what I mean by that is that companies that are at the top tend to make anti customer decisions because they have lost the concept that pleasing customers matters as priority one.
I think it's pretty dishonest of Anthropic to frame their watermark as EU regulation compliance. The EU regulation, from my understanding, requires AI content to be labeled for human viewers. In the meanwhile the Anthropic new release on the watermark says this.
> The difference between watermarked and un-watermarked text will not be distinguishable to readers
https://www.anthropic.com/news/claude-text-watermark
Which is to say, it does not actually meet the EU AI act requirements which require transparency to humans. Not to mention that if the detection requires access to the base models, it makes anthropic the only entity who gets the say on if a piece of text comes out of Claude. Anthropic is both the player and the referee here.
If there is one takeaway you should have from this fiasco it is that you should be wary of using tools that doesn't serve your needs and your needs only.
Lots of faux outrage, rambling and hyperbole here from Gruber.
“Absurdly and insultingly”? Come on…
Reminder: your favorite distilled model does not treat you, the customer, as an adversary and mess with your output.... May the free market win.
I don’t understand how this works for anything but prose. Is that the point? In any code or structured output, there just isn’t the flexibility, and depending on how the user requests the output be constrained there is even less (“answer only True or False”). So is it just chat responses? If I ask the API to tell me a story about Alice and Bob then it watermarks it, but when I ask it some implausibly constrained thing like write a story about Alice and Bob with each word starting in rotation with the letters alicebob, does it try to do so and hope there are roughly équiprobable tokens regularly?
gruber is really out of his element with ai commentary, I fully support the general skepticism but he's seemingly arguing against something he doesn't quite grasp
I'll be honest, who fucking cares? Why would you use AI to write for you and then complain that people know AI wrote the code?? If you know people wouldn't like it, why even try!?
The objection seems to be that Claude will always write worse prose than a human writer, even if the writing Claude generates is understandable.
Yeah, John. We're all OK with that.
Anthropic should just pull out of the EU at this point. Europeans who really want to use it can VPN, and if they did they wouldn’t be able to hide behind their various comical tech laws.
Nanny state nonsense indeed.
As a non American/non European resident all I see from the Europeans are 0 contributions to software progress at any large scale while they surely do a lot of crying and huffing and puffing and demanding. Lots of complaining and rule making but not a lot of creating is a bad look.
> the only acceptable answer for why an LLM should choose bananas instead of pineapple (or coconut, or guava, or papaya...) is that it has determined that it’s the best fit for the intended meaning, tone, and sentiment of the text.
It already fails. It randomly picks between close candidates. To help fool people into believing in intelligence claim, I guess.
"Anthropic's . . . Claude is a Perversion of Writing."
FITFY.
I have no sympathy for writers whining about what the AI is doing to 'their' writing. It's only your writing when you write it. There's any easy way to avoid this: don't fucking use it. Use you own brain.
I keep seeing an irritating misconception in this space, which is that the alternatives chosen by these algorithms are supposed to mean the same things as what they're displacing. That's not true, and not how LLM generation works. Complaints that two different choices don't mean the same thing miss the entire point.
[flagged]
Gruber has a ridiculous knee-jerk response to anything the EU does, so hardly a surprise he didn't come to the table with a sober facts-based response.
Not telling someone you used AI is a perversion of writing. Also agree that an AI proofreader should not claim authorship, but in most other cases, the AI is not reading your mind, it's only watermarking its own usage, and we kind of need more of that.
> I want any LLM I use to choose the very best, most precise words at every single decision point.
Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo random. The output it generates is one of the possible outputs it would have generated before, just now it's deterministic and will generate the same thing every time.