logoalt Hacker News

jay_kyburzyesterday at 9:31 PM1 replyview on HN

I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.

You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.


Replies

jay_kyburzyesterday at 9:34 PM

How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.

show 1 reply