alt Hacker NewsQuestion: do these bots not respect robots.txt?
I haven't added these scrapers to my robots.txt on the sites I work on yet because I haven't seen any problems. I would run something like this on my own websites, but I can't see selling my clients on running this on their websites.
The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an IP visits that page, they get added to a blocklist which simply drops their connections without response for 24 hours.
Replies
You haven't seen any problems because you created a solution to the problem!
> Question: do these bots not respect robots.txt?
No they don't, because there is no potential legal liability for not respecting that file in most countries.
> The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an IP visits that page, they get added to a blocklist which simply drops their connections without response for 24 hours.
I love this idea!