alt Hacker NewsIt's not the "same" crawler. Probably each thread or each cluster machine instance of the crawler hitting it independently.
Replies
mrweasel • today at 1:58 PM
I sincerely doubt that search engines run their crawlers on a single machine and they got it figured out.
That's still the same crawler system though. And it's lazy engineering to not build in something to track when you last requested a url.
And it's quite a trivial feature at that.