On top of that, the user paid for those tokens, so if there is an owner, it should be the user, not the provider.

I think they are doing us a disservice by perpetuating this focus on intellectual "property" [1] in regard to AI.

The real issue with Anthropic, OpenAI etc. is not that they have used all of our public knowledge for training their LLMs. Creating new work from old and learning from prior generations is what we all do. The issue is that they want to claim all of the benefits for themselves. They are standing on the shoulders of giants and have contributed an inch themselves, yet want to privatize the power of the whole giant. We shouldn't let them "own" these models.

The influence on society by AI is so novel that it's reasonable to craft new laws specifically for them. There are a lot of ways to deal with their power grab. We could force them to open source the models after two years. Or we could tax tokens or compute. We just need to agree that the power grab is the problem, the privatization of our cumulative knowledge, and not some details about copyright infringement.

[1] I know I'm going to risk dissent just by putting quotation marks here. But I think for this topic specifically it is crucial to understand that intellectual property is an arbitrary social/legal construct. With physical stuff, there is an inherent scarcity. If you steal my smartphone, I no longer have it. If you steal the character from my book, I... have a harder time selling my next book? Our ancestors have invented copyright to solve a specific problem, but the solution has become perverted over time. There are a lot of egregious cases out there (looking at you, Disney), but even relatively tame success cases don't look good. Society has paid J.K. Rowling a literal billion for her work and still this cultural touchstone of a generation remains privatized. Imagine what other authors could have build upon her stories, if only they were allowed to publish their own stories with these characters. She has not been a particularly good steward in the past decades.

Even copyrighted information can never be "stolen". It can only copied without authorization.

If I hire you for a consulting project, do I own the screen recording of your computer while you completed it?

they also can't claim to be the owner of the output either because they can't claim to have had a valid license for the inputs.

[dead]

While I feel it is morally OK for users to access the metadata of their own exchanges, there's something awry with that formulation, (not-)"stealing" is about more than just copyright or ToS.

For example, consider my browser cookies that authenticate me to HN right now. Nobody even wants to copyright them, but if you were to somehow acquire a copy I'd very much consider it "stealing."