alt Hacker NewsDoes crawl-to-refer mean that for every 40k pages ClaudeBot crawls, only 1 outbound link is clicked from it?
Replies
johng • yesterday at 4:22 PM
Yes, that's exactly what it means.
Does crawl-to-refer mean that for every 40k pages ClaudeBot crawls, only 1 outbound link is clicked from it?
Yes, that's exactly what it means.
Claude has an order of magnitude fewer users on its web product while training models that are just as large and advanced as OpenAI, so this makes sense.