logoalt Hacker News

grajaganDevlast Thursday at 2:23 PM1 replyview on HN

This keeps generating new pages to keep the crawler occupied.

Looks like this would tarpit any web crawler.


Replies

BryantDlast Thursday at 2:29 PM

It would indeed. Note the warning: "There is not currently a way to differentiate between web crawlers that are indexing sites for search purposes, vs crawlers that are training AI models. ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS."

show 3 replies