I just put Anubis in front of my self-hosted forge this morning because AmazonBot had helped itself to 750 GiB (!) of traffic to my public repos this month!
At least, it claimed to be AmazonBot…
Are they in this space? [1] One could map the ranges into a web daemon and rate limit them or just 'ip route add blackhole ${cidr}' each cidr block.
[1] - https://ip-ranges.amazonaws.com/ip-ranges.json
Do you have a robots.txt?
Are they in this space? [1] One could map the ranges into a web daemon and rate limit them or just 'ip route add blackhole ${cidr}' each cidr block.
[1] - https://ip-ranges.amazonaws.com/ip-ranges.json