>absurd amount of effort every website is pouring into anti-scraping tech.

Maybe some people do not want to provide their entire website as free training material to [insert LLM lab here]?