alt Hacker NewsI have always blamed Chinese AI operators for the hundreds of thousands of residential proxy requests I receive each day, but there's no way to be sure.
They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).
Replies
Identified AI bots that finally made me cave and put anubis (with very liberal limits) up were anthropic/openai/perplexity according to the headers they sent. They were also over 99% of traffic despite proper robots.txt (site only has a few hundredth visits per day), which they ignore. I don't mind bots or crawlers, but these bots got the same page/query string every time they decided it was time for a crawl, which was about daily, cache control, last modified headers they also never seemed to be interested in. They were very very far from well behaved, unlike search engine crawls, and impossible to tame.
Yeah, but how these residential proxies get in the residential areas? I would guess popular VPN providers are mostly behind this. They sell you VPN and sell bots residential proxy access on the other end.
If there’s no way to be sure, then why blame Chinese operators? Because American companies have given you so many reasons to believe they’re acting in good faith?
I don’t particularly trust anyone at this point, and I assume AI traffic on my web servers are all malicious actors, and never bothered to consider what country the malicious bots came from. Mostly because I couldn’t possibly care less, but it kept happening even with geofencing and rate limits, so some American companies who own domestic IP blocks are facilitating them.