alt Hacker NewsMy experience disagrees with the 'Respects robots.txt' column for most of the bots listed. Would love to see more details of how they determine that metric.
My experience disagrees with the 'Respects robots.txt' column for most of the bots listed. Would love to see more details of how they determine that metric.
Are you verifying the IP, or just blindly trusting the user agent?