alt Hacker NewsI was under the impression that they do obey robots.txt now? There are clearly a lot of dumb agents that don’t, but didn’t think it was the major AI labs.
I was under the impression that they do obey robots.txt now? There are clearly a lot of dumb agents that don’t, but didn’t think it was the major AI labs.
After 3 years of pirating and scraping the entire world by doing the above, I guess they have everything that they now need or want.
So then it's better to start obeying ROBOTS.txt as a ladder pull through a "nicely behaved" image advantage.