One would assume legit spiders obey robots.txt.
The real problem is most of them just pretend to be normal users now robots.txt only works if they choose to care.
This, to me, is the strongest argument to offer these slop generators. It provides an incentive to follow the robots.txt.
The real problem is most of them just pretend to be normal users now robots.txt only works if they choose to care.