logoalt Hacker News

stickfigure • today at 2:06 PM • 2 replies • view on HN

> It's wrong to ignore a robots.txt.

It's not nice. It is, however, not illegal.

And before you agitate against that, think deeply about the consequences. A world where website owners can legally dictate exactly what software you use to access their content.


Replies

trollbridge • today at 6:58 PM

The copyright terms on my own site make ignoring it and scraping it expressly illegal.

Unless copyright law has been entirely repealed?

➕ show 1 reply
BizarroLand • today at 4:22 PM

They didn't say illegal, they said wrong. And they were right. You even agree with them by saying "It's not nice". Non-nice things are wrong in some way.

And, we already live in a world where website owners can legally dictate exactly what software you use to access their content. They can require that you log in with specific browsers running specific software on your computer, they way online schools run exam proxies.

➕ show 1 reply