alt Hacker Newsyou mean AI crawlers from Microsoft, owners of Github?
Replies
haiku2077 • today at 12:32 AM
The big companies tend to respect robots.txt. The problem is other, unscrupulous actors use fake user agents and residential IPs and don't respect robots.txt or act reasonably.
➕ show 1 reply
I have no idea where they are from. I'd surprised if MS is using a network of 1M+ residential IP addresses, but they've surprised me before ...