logoalt Hacker News

charcircuittoday at 3:45 AM0 repliesview on HN

>Honors robots.txt

Is it possible to ignore robot.txt in the case the crawl was triggered by a human?