logoalt Hacker News

xg15today at 1:26 PM1 replyview on HN

Facebook just decided that instead of loading the robots.txt for every host they intend to crawl, they'll just ignore all the other robots.txt files and then access this one a million times to restore the average.


Replies

Vinnltoday at 2:56 PM

Ah yes, robots_georg.txt.