They know that the rules apply to them. They hope that they can avoid being caught.

It’s only bad if you’re a closed, for-profit entity

</sarcasm>

You nailed it.

For what it's worth, the big AI companies do have opt out mechanisms for scraping and search.

OpenAI documents how to opt out of scraping here: https://developers.openai.com/api/docs/bots

Anthropic documents how to opt out of scraping here: https://privacy.claude.com/en/articles/8896518-does-anthropi...

I'm not sure if Gemini lets you opt out without also delisting you from Google search rankings.