If it’s been this way since February, how have AI crawlers not “caught up” yet?
The internet is big, but it isn’t that big. I’d expect to see a sudden dropoff as they start re-checking content that hasn’t changed, with some sort of exponential backoff.
Instead, my takeaway is that they are AI crawlers aren’t indexing to store in a way we’re used to with typical search engines, and unilaterally blocking these crawlers across the board would result in quite the “effect”.