logoalt Hacker News

dormentotoday at 1:56 PM2 repliesview on HN

Has anyone done research on the topic of trying to block these bots by claiming to host illegal material or talking about certain topics? I mean having a few entries in your robots like "/kill-president", "/illegal-music-downloads", "/casino-lucky-tiger-777" etc.


Replies

pousadatoday at 2:05 PM

Yea I can’t see how that could backfire in any way

DetroitThrowtoday at 2:59 PM

FB crawler is used for national security reasons at times. The first would probably make it more active.