> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it.

Kinda sounds like we're missing a peer to peer network here.

Instead of downloading the same data over and over again we can just download it once and then share it.

Wouldn't that be better for everyone involved?

It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)