Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.
Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.