alt Hacker NewsYou can still ban everyone who doesn’t respect robots.txt. My experience with a crawler is that people don’t actually provide a robots.txt far more often. I can back off on 429 but I’d rather use Crawl-Delay etc.
Bots have limited ability to announce themselves. I just use fixed IP range and UA. But the Chinese trawlerbots use smart TV residential proxies and will copy someone else’s UA.
Replies
> My experience with a crawler is that people don’t actually provide a robots.txt far more often.
I think many don't see the point of robots.txt as it will just be ignored by most bots anyway, so instead concentrate on more complicated measures.
Those would fall under the "stealth" bots subject to a fine.
The whole point is that if you can prove a company is doing this kind of shit, they should be on the hook for it. Of course, I'm not expecting any company in China to actually pay the fines, but this opens up some options such as seizing foreign assets.
> trawlerbots use smart TV residential proxies and will copy someone else’s UA.
What bugs me the most are the ones that cycle through a list of LLM user agents.. What are you actually? You can't be Baidu and Bing at the same time.