alt Hacker Newsthinking running a website never got you 99.9% of requests from bot is just delusional.
anyway, if you really want to block scrappers, give up Anubis. it just about real users and do nothing against bots. simple block ISN that belong to cloud providers. it's that simple.
Replies
threetonesun • today at 5:36 PM
Some of us deployed websites before bots. Heck once upon a time you had to ask nicely for them to even come visit.
I got to watch traffic shift from cloud IPs to residential proxies in real time after I blocked the cloud IPs. It’s endless whack-a-mole.
What really pisses me off was Meta’s bots ignoring robots.txt when I tried to prevent high levels of traffic hitting meaningless URL variants on a Mediawiki site. Everybody else stopped hitting those URLs but not Meta which was flooding me with so much traffic it was impacting site performance. There source IPs and user agents are now banned.