alt Hacker NewsOnly tangentially related, but I've recently realized that unless you're some sort of business with a valid requirement of that, you can just block incoming traffic from all of AWS, DigitalOcean, Azure, etc.
I think the spark came from some other HN comment I saw? So thank you for that, whoever wrote that.
That alone is not a sufficient strategy of course, but you can get rid of a lot of bots by just refusing to talk to where they tend to live.
Of course, if everyone were to do that, they'd eventually pivot to residential proxies. But I don't see that happening.
Replies
I initially thought this was a good idea too until another comment pointed out you'll lose most business or personal customers in offices because they connect through VPN gateways running on those cloud services (especially Azure)
Allmost all the bottraffic I see is from random residential proxies though.
I currently tarpit traffic from AWS, Azure, GCP, etc. since it's not like any of my real customers have their PC inside those networks. (I then use that for experiments with providing bogus data to AI scrapers.)
Doesn't do much to help with the "VPN that uses random consumer PCs and phones" traffic.
If you block datacenter IP you’re also going to block all of the legitimate search engine crawlers.
Won't this harm SEO? What about agentic software suites where authorized users should be able to drop an app passkey into grokbot or such?
>But I don't see that happening.
Then you are not looking. Already have to switch to resi proxies from time to time when encountering yet another smartass who decided to play security theater and block datacenter ips on his shitpile he calls website, because "OUR USERS MUST BE GOOD BOYS WHO LOVE TO LIVE UNDER SURVILANCE AND ALLOW ISP TO TRACK THEM 24/7" (and before you say something like "others can track you too!", your ISP has the ability to knock on your door, they should be given as little information as possible, let them do metadata analysis on per-person basis if they really want it, but no 1-click mass surveillance shit).
If you are valuable, you will be be scraped. Turning user hostile in a hopeless endeavor to "protect yourself" is not the solution.
There have been a number of articles on the frontpage of HN about the residential proxy traffic. It is happening. This was one of the most popular:
https://news.ycombinator.com/item?id=49491791