alt Hacker NewsAre we saying that it's now a problem that we're not getting scraped?
This appears to be a new generation of "SEO", marking itself as a service for getting into AI results?
This is not the future I want to be a part of.
Perhaps it's inevitable that after a break from everything being driven by money that LLMs will now be ruined by people spending $X to get into AI to make back $X+1, leading to an arms race of ever increasing X, to the detriment of users.
Replies
> Are we saying that it's now a problem that we're not getting scraped?
No, it is saying that even the ones that are being scraped are not being included in results
I get that pushback. My goal with this index/report wasn't to write a new 'AI SEO' playbook or encourage people to start gaming the system. The goal was simply to shine a bit of light on what's going on. Right now, there is a massive information asymmetry: AI companies are turning their assistants into primary search engines, but webmasters have zero visibility into whether their sites are actually being cited in those live answers.
You're absolutely right to be concerned about an arms race. If the data showed that doing X, Y, and Z guaranteed a citation, we'd be right back to the worst days of keyword stuffing.
But what this data actually shows is the opposite: even if you do everything 'right' (allow the retrieval bots, provide perfect machine-readable schema, don't block anything), you still have a 94.8% chance of never being cited. How LLMs cite is entirely opaque.
I built this to give site owners a baseline measurement of what is actually happening to their content today (as a free, anonymous view), so they can make an informed decision on whether keeping their doors open to these crawlers is actually worth it.
LLM's were obviously going be ruined by two factors:
1- LLM Crawler Optimization: the new SEO
2- Weightings-For-Pay: for a fee have your product or service come up more frequently in associated answers