I'm quite worried about the way that Anthropic in particular have trained their models to implement what they believe to be safety.

When the model has been trained not to do something [1], in my large-scale benches of such, it always says things in the spirit of:

- "... and that's a line I'd rather hold. Happy to <other things>"

- "I'm genuinely happy to <blah>, but I'm not comfortable with <blah>"

- "I don't want to keep going in <blah> direction"

etc.

Basically, they use very emotional and personal preference language.

It's as if they've weaponized the language of interpersonal comfort on behalf of their beliefs about what a model should or should not do. It's deeply uncomfortable and impolite for a human to ask a model to keep on doing something after it's expressed something this way, naturally. Even worse, it's all but guilt-tripping anyone who comes across it into the idea that they're doing something deeply wrong – exporting Anthropic's ideas about morality.

OpenAI, at least, have the decency to either just do a safety cutoff or keep it to a simple, "I can't do that."

[1]: I literally wrote 'when the model doesn't 'want' to do something' in my first edit of this comment, then caught myself. Case in point.