This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt.
Exactly. You disobey robots file => we'll make your crawl gain a net negative.
alt Hacker News
Exactly. You disobey robots file => we'll make your crawl gain a net negative.