alt Hacker NewsIf the scrapers are going to get it anyway, put the data up as a zip somewhere.
Replies
antisthenes • yesterday at 7:46 PM
> Read the Docs, a non-profit that hosts documentation for open-source software, who watched a single crawler download 73 terabytes of zipped HTML in one month, costing it over $5,000 in bandwidth
From the article.
Not the same site, but an example of the same issue.
It would not make any difference.