logoalt Hacker News

Crawling a billion web pages in just over 24 hours, in 2025

16 pointsby pseudolustoday at 3:54 AM1 commentview on HN

Comments

finnlabtoday at 7:49 AM

Nice work, but I feel like it's not required to use AWS for this. There are small hosting companies with specialized servers (50gbit shared medium for under 10$), you could probably do this under 100$ with some optimization.