alt Hacker NewsLet's not conflate crawlers with the traffic that bot protection services block. A crawler that respects robots.txt is a good internet citizen and can provide a vital service.
Replies
sreekanth850 • today at 2:34 PM
And you think all this web AI crawlers will respect robots.txt. That era is gone.
However, so-called AI crawlers are not the same as crawlers of yore. They hit live pages every time a user prompt triggers a web search.
This Web Search API, unlike an AI crawler, only fetches periodically. It feels like a step in the right direction for managing resource strain across the internet. If only the LLM giants could do something similar.