> It's wrong to ignore a robots.txt.
It's not nice. It is, however, not illegal.
And before you agitate against that, think deeply about the consequences. A world where website owners can legally dictate exactly what software you use to access their content.
They didn't say illegal, they said wrong. And they were right. You even agree with them by saying "It's not nice". Non-nice things are wrong in some way.
And, we already live in a world where website owners can legally dictate exactly what software you use to access their content. They can require that you log in with specific browsers running specific software on your computer, they way online schools run exam proxies.
The copyright terms on my own site make ignoring it and scraping it expressly illegal.
Unless copyright law has been entirely repealed?