alt Hacker NewsThe scrapers should use some discretion. There are some rather obvious optimizations. Content that is not changing is less likely to change in the future.
The scrapers should use some discretion. There are some rather obvious optimizations. Content that is not changing is less likely to change in the future.
They don't care. It's the reason they ignore robots.txt and change up their useragents when you specifically block them.