logoalt Hacker News

tgvlast Thursday at 3:46 PM1 replyview on HN

You can't make money out of studying robots.txt, but you can avoid costs skipping bad web sites.


Replies

xeromallast Thursday at 8:06 PM

Sounds like a benefit for the site owner. lol. It accomplished what they wanted.