Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

This is a weird complaint.

Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia.

There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site?

Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most sites deserve links because they were crawled seems like a statistical fallacy. You could say the same thing about the percent of sites crawled by Google versus ever showing up on first page of results.

Are we saying that it's now a problem that we're not getting scraped?

This appears to be a new generation of "SEO", marking itself as a service for getting into AI results?

This is not the future I want to be a part of.

Perhaps it's inevitable that after a break from everything being driven by money that LLMs will now be ruined by people spending $X to get into AI to make back $X+1, leading to an arms race of ever increasing X, to the detriment of users.

This makes sense. A handful of websites hold most of the “trusted” info because they’re massive. I don’t expect you to quote my blog with only three entries. The real trick would be getting AI companies to stop hammering sites that don’t show up in answers, but even if they don’t use a source for an answer, crawling still provides value.

Odd results for me. Last month I tested five AI chat sites, with web search turned off, and four of them had a shadow of information about me as a person (what kind of books I write, what tech I use, and a random bit of other information). The linked site gave me a zero score because it was testing if the AI models recommended my site for business or sales queries.

Makes me wonder if paid Medium and paid news are input to AI training. Surely, they are.

This makes sense. Why would everyone be coted for the same answer

This is why we need the HTTP 402 standard to become common.

If websites charge pennies per AI crawl, they will make more money than ever being reference in that 6.2% of websites that get cited (of which even another small percent get any follow through that leads to a sale or ad click)

HTTP 402 also basically extends the pay per token model people have gotten used to with AI model providers, except applied to the whole web, with the added privacy benefit in that there is no need for sellers of content to “know their customer”, and indeed it may even be impossible to do so because of how the payment gateways operate.

> 94.8%

This seems consistent with Sturgeon's Law.

Call it web 4.0

Blocking known bot identifiers via robots.txt does nothing by the way. Too many labs are running sneaky crawlers that do not respect robots.txt. You will need to take extreme measures: blocking basically all datacenter IP ranges, VPN IPs, aggressive rate limiting, etc.

Blocking LLM crawlers has become the number one use case for our IP database customers at https://focsec.com/

We don't have a page rank mechanism. Most likely people are paying or threatening AI companies to boost certain sources as authoritative

[dead]

[flagged]

It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a legal or medical opinion supported by research.