logoalt Hacker News

apples_orangestoday at 9:35 AM1 replyview on HN

HTTP 412 would be better I guess..


Replies

jsheardtoday at 12:00 PM

You shouldn't really serve aggressive scrapers any kind of error or otherwise unusual response, because they'll just take that as a signal to try again with a different IP address or user agent, or a residential proxy, or a headless browser, or whatever else. There's no obligation to be polite to rude guests, give them a 200 OK containing the output of a Markov chain trained on the Bee Movie script instead.

show 1 reply