alt Hacker NewsAfter 3 years of pirating and scraping the entire world by doing the above, I guess they have everything that they now need or want.
So then it's better to start obeying ROBOTS.txt as a ladder pull through a "nicely behaved" image advantage.
After 3 years of pirating and scraping the entire world by doing the above, I guess they have everything that they now need or want.
So then it's better to start obeying ROBOTS.txt as a ladder pull through a "nicely behaved" image advantage.
Obeying robots.txt (now) is still better than not obeying it, regardless of what they did before.
The alternative is to say that bugs shouldn’t be fixed because it’s a ladder pull or something. But that’s crazy. What’s the point of complaining if not to get people to fix things?