logoalt Hacker News

hoistbypetardyesterday at 4:23 PM0 repliesview on HN

If you do that and don't follow robots.txt, you are blocked. If you do that and follow robots.txt, fine. That's all we wanted you to do anyway. Just follow the instructions that well-behaved scrapers are meant to follow.