alt Hacker NewsPer https://thelibre.news/foss-infrastructure-is-under-attack-by..., all of the major American AI companies are not respecting robot.txt and participating in the AI-fueled DDoS of the internet.
Per https://thelibre.news/foss-infrastructure-is-under-attack-by..., all of the major American AI companies are not respecting robot.txt and participating in the AI-fueled DDoS of the internet.
The issue is that UA are editable by the user, and there is no proof that some random person/scraper isn't just using a suspected trusted bot's UA string. Every ethical service also posts what IP addresses they use, so that people can compare the traffic they get to see if it is actually their bot scraping. What this article describes is the game of every third-party unethical scraper; they do anything and everything to try and get their request through. They steal UA's, they steal residential IP addresses through botnets, they attempt to circumvent CAPTCHAs using AI, etc. So the behavior in this article is not prove for any major AI provider doing unethical scraping.