alt Hacker NewsAssuming that even works from a researcher's perspective, it's working back from a specific goal. There's 0 actual instances (and I've been looking) where verbatim code has been spat out.
It's a convenient criticism of LLMs, but a wrong one. We need to do better.
Replies
thechao • today at 1:11 PM
I don't know code examples, but this tracks, for me. Anytime I have an agent write something "obvious" and crazy hard -- say a new compiler for a new language? Golden. I ask it to write a fairly simple stack invariant version of an old algorithm using a novel representation (topology) using a novel construction (free module) ... zip. It's 200loc, and after 20+ attempts, I've given up.
> There's 0 actual instances (and I've been looking) where verbatim code has been spat out.
That’s not true. I’ve seen it happen and remember reports where it was obvious it happened (and trivial to verify) because the LLM reproduced the comments with source information.
Either way, plagiarism doesn’t require one to copy 100% verbatim (otherwise every plagiarist would easily be off the hook). It still counts as plagiarism if you move a space or rename a variable.
https://xcancel.com/DocSparse/status/1581461734665367554
https://xcancel.com/mitsuhiko/status/1410886329924194309
> We need to do better.
I agree. We have to start by not dismissing valid criticisms by appealing to irrelevant technicalities which don’t excuse anything.