alt Hacker NewsNo, the original "they're made out of meat" works because we're confident that we are in fact intelligent and conscious, despite how ridiculous and unlikely the author manages to make it sound.
"They're made out of weights" works precisely because LLMs really do have this mysterious property that they seem somehow intelligent even though nobody can explain exactly why, and there's active debate over whether they could be considered conscious.
The thing being discussed isn't simply an arbitrary MacGuffin; in both cases the nature of the thing is central to the impact of the story.
I disagree; it works in the original because it's the unlikely consciousness that produces the text itself; in the LLM case, it's produced by the likely consciousness.
"Imagine how other intelligences would view us", written by us, hits a lot less hard when it's "imagine how our intelligences view a thing we are claiming is intelligent", not written by it.