> The best and most useful data is often inaccessible to crawlers.

Interesting point.

> ost open forums are dead because discussions happen on login platforms like Reddit, Facebook, Instagram, X, Discord, etc

Ironically isn't one of the reasons some of those platforms started to use logins was so they could track users and better sell their information to ad people?

Obviously now there are other reasons as well - regulation, age verification etc.

Does this suggest that the AI/ad platforms need to tweak their economic model to share more of the revenue with content creators?