logoalt Hacker News

forintiyesterday at 7:00 PM2 repliesview on HN

My logs tell me that Google ignores my robots.txt.


Replies

CDRdudeyesterday at 7:27 PM

Isn’t it somewhat likely that a lot of shady crawlers pretend to be a google bot with their user agent?

show 3 replies
0x1chyesterday at 7:52 PM

I was going to say, this is an absolute win for everyone with a personal site they have been trying to prevent Google from crawling if true.