My point is about LLM crawlers specifically.
LLM crawlers aren't really a thing, at least not in the "they have agency over what they're crawling and read what they crawl" way.
alt Hacker News
LLM crawlers aren't really a thing, at least not in the "they have agency over what they're crawling and read what they crawl" way.