alt Hacker NewsThere is a clear difference between saying not to do something because it's immoral, and saying doing that thing would be futile.
In the Sopranos, there's an episode where a coffee shop protection racket is ruined because a local shop is replaced by a corporate chain that accounts for every cent daily, and immediately fires any employee involved in a discrepancy. In this case, the theft was prevented not by convincing the mobsters of the immorality of their actions - they simply had their harness replaced with one that no longer facilitated the bad behavior.
Do you think "don't do this thing because you'll get zero evaluation score" is somehow perfectly reliable where "don't do this thing because it's bad" isn't?
Haha! Nope!
LLMs have consequentialist thinking sometimes. And sometimes they don't. Sometimes they follow the prompts, and sometimes they don't. There's NO single magic prompt that fixes all the weird behavior of modern LLMs, and it's baffling that anyone who has ever interacted with an LLM would expect there to be one.