rate limit per ip that progressively keeps decreasing req/mins every few mins?
What if scrapers ips are millions of smartphones? If I was as evil as an AI scraper company that is not obeying robots.txt I would totally build/buy thousands of small games/apps for mobiles to use them as jumphosts to scape the web. This is probably happening already.
so why not just do that for these scrapers, rather than complicate it by encrypting and decrypting, which is just obfuscation as the private key is clearly available to the end-user?
Just skip the whole encryption/decryption shebang then.
But no, this does not work, scrapers are using residential ips, and they have enough that they can rotate between them if they get blocked.