logoalt Hacker News

skybriantoday at 12:55 AM0 repliesview on HN

If two customers crawl the same website and it uses crawl-delay, how does it handle that? Are they independent, or does each one run half as fast?