alt Hacker NewsMy point is that if crawlers have to worry about poison that may make them start to respect robots.txt or something. It's a bit like a "Beware of Dog" sign.
My point is that if crawlers have to worry about poison that may make them start to respect robots.txt or something. It's a bit like a "Beware of Dog" sign.
How would that become a strong, stable signal, if both highly valuable and highly slopified content will use robots.txt?