logoalt Hacker News

kraphtyesterday at 7:17 PM0 repliesview on HN

His example of a web crawler stinks. I can get behind the philosophy, but his specific web crawler example at a minimum needs to retry, respect robots.txt, and rate limit per domain. So no, xargs + curl is a dumb example.