logoalt Hacker News

plaidfuji12/08/20243 repliesview on HN

Do you just pay the bill for the resources indefinitely?


Replies

hansvm12/08/2024

I'm not the person you're asking, but I maintain a number of scraping projects. The bills are negligible for almost everything. A single $3/mo VPS can easily handle 1M QPS (enough for all the small projects put together), and most of these projects only accumulate O(10GB)/yr.

Doing something like grabbing hourly updates of the inventory of every item in every Target store is a bit more involved, and you'll rapidly accumulate proxy/IP/storage/... costs, but 99% of these projects have more valuable data at a lesser scale, and it's absolutely worth continuing them on average.

NavinF12/09/2024

Inbound data is typically free on cloud VMs. CPU/RAM usage is also small unless you use chromedriver and scrape using an entire browser with graphics rendered on CPU. We're taking $5/mo for most scraping projects

mNovak12/09/2024

I paying < $0.50 a month, and that's primarily driven by S3. For the scraping itself I'm using lambda, with maybe minutes of runtime per day.