alt Hacker News>You could go through and change all the points in the original story to be about a toaster instead. It would require you to edit text, but there is no barrier to doing that.
Yes, there very much is a huge barrier. The copy and the original both keep the same subject matter: intelligence/human-like behavior.
The toaster doesn't. We could do edits, and the story (original or copy) would lose all its potency.
The surprise is "but how is the richness of intelligent behavior produced from something as basic as meat/weights". Why is kind of surprising reductionism is both cases.
Whereas nobody is surprised that "a metal strip with electricity flowing through it from a power source" heats pieces of bread to a specific temperature. Even if a toaster was "metal strips" all the way down, it's nowhere near as impressive jump from substrate to behavior, nor is the behavior as important to us and touches the core of our existance.
I think you're misunderstanding me. There's no barrier to writing this story but about a toaster: you literally just type different words. Writing a story in which a toaster in conscious is not even new [1].
That is entirely separate to whether or not it's meaningful to write a story about a conscious toaster. Again, expressing an idea in fiction is not related to the accuracy of that idea.
What makes the original story interesting is that it was written by the thing claiming to be conscious, which is what takes it from 'a story' to 'a story making an important point'. That's not the case with a hypothetical story about a toaster, and it's not the case with this story about LLMs.
The story is convincing because it's well-written, not because it is factual.
[1] https://en.wikipedia.org/wiki/The_Brave_Little_Toaster