alt Hacker NewsI hope you leave it on the WAF. If they're only just deciding to respect robots.txt, which has been internet infrastructure forever, then it's probably still incredibly amateur software with 'Amazon-priorities' rather than 'responsible internet traffic' priorities.
The responsible internet is dead. Every big actor on the internet is selfish now that there's money involved. And has been for 20 years.
Google only respected it because blocking Google from crawling your site used to hurt you more than it hurt Google.