logoalt Hacker News

greycol • today at 9:21 PM • 0 replies • view on HN

US case law may not be on your side there. Generally unless you require a login to access the site (and even then you can only really ban for terms of service rather than recover damages unless they hacked around login requirements) judges have found that using scrappers on publicly served data is fair use. It gets murkier with creative content (rather than straight data) in what they can do with it after scrapping but the scrapping itself would still be 'legal' in the they can do it without judicial consequence sense.

It's one of the reasons why sites that served a lot of data used to have APIs or back channels to get that data even if they technically made money by serving ads on pages of that data, because they were going to get scrapped no matter what, so having a legitimate way to get data that behaved nicely with the servers and was nicely readable by requestors was win-win or at least not lose-lose.