alt Hacker NewsNoob question, I recall https://news.ycombinator.com/item?id=49491791 mentioned how the kernel source code and diffs were being crawled excessively. All else being equal, won't all the AI crawling agents mean traffic to docs should at least be stable(presuming changes happen all the time)?
Replies
simonw • today at 2:30 PM
A crawler visiting your docs won't help a human decide to pay for your commercial service advertised on those docs.
ghm2199 • today at 2:18 PM
And also the fact that agents are solving anubis level 5 challenges Tailwind would not be able to figure out how much traffic is from people vs bots. So that would mean the traffic should be up...
Tailwind is so pervasive it's 100% in pretraining _and_ RLHF.
I've done several Tailwind projects and rarely did models from various providers need to reference Tailwind documentation.