It's a similar loophole as public libraries. When I was a kid, I read thousands of books from the library, without paying anyone anything.
But as for the crawl loophole: CCBot obeys robots.txt, and CCBot also preserves all robots.txt and REPL signals so that downstream users can find out if a website intended to block them at crawl time.
alt Hacker News
It's a similar loophole as public libraries. When I was a kid, I read thousands of books from the library, without paying anyone anything.
But as for the crawl loophole: CCBot obeys robots.txt, and CCBot also preserves all robots.txt and REPL signals so that downstream users can find out if a website intended to block them at crawl time.