logoalt Hacker News

jsheardtoday at 12:00 PM1 replyview on HN

You shouldn't really serve aggressive scrapers any kind of error or otherwise unusual response, because they'll just take that as a signal to try again with a different IP address or user agent, or a residential proxy, or a headless browser, or whatever else. There's no obligation to be polite to rude guests, give them a 200 OK containing the output of a Markov chain trained on the Bee Movie script instead.


Replies

loloquwowndueotoday at 4:22 PM

Unless your output is static, you’d then be paying the cost of running the markov generator.