My logs tell me that Google ignores my robots.txt.
Isn’t it somewhat likely that a lot of shady crawlers pretend to be a google bot with their user agent?
I was going to say, this is an absolute win for everyone with a personal site they have been trying to prevent Google from crawling if true.
Isn’t it somewhat likely that a lot of shady crawlers pretend to be a google bot with their user agent?