logoalt Hacker News

vivzkestrellast Friday at 10:27 AM3 repliesview on HN

rate limit per ip that progressively keeps decreasing req/mins every few mins?


Replies

63stacklast Friday at 8:28 PM

Just skip the whole encryption/decryption shebang then.

But no, this does not work, scrapers are using residential ips, and they have enough that they can rotate between them if they get blocked.

prmoustachelast Friday at 12:28 PM

What if scrapers ips are millions of smartphones? If I was as evil as an AI scraper company that is not obeying robots.txt I would totally build/buy thousands of small games/apps for mobiles to use them as jumphosts to scape the web. This is probably happening already.

show 1 reply
chiilast Friday at 12:10 PM

so why not just do that for these scrapers, rather than complicate it by encrypting and decrypting, which is just obfuscation as the private key is clearly available to the end-user?

show 1 reply