alt Hacker NewsI've been working on a site. It's new, domain is only a few weeks old. It's got SSL, so all the bots know it exists. It's never had any sub-pages exposed, just the placeholder lander, no links.
Somehow in Google search one of the unguessable pages is indexed. We have used Claude and Gemini to assist with some design aspects.
I'm thinking some aggressive data ingestion/indexing is happening by all the bots in the quest for frontier models.
Replies
Are you using Cloudflare by any chance? I think the Crawler Hints setting [1] exposed some of my "secret" pages in the past.
[1] https://developers.cloudflare.com/cache/advanced-configurati...
Depending on the CMS, if it's wordpress (15% chance, ha) there is a sitemap function built-in out of the box. The bots don't need to guess.
There's a couple avenues besides just stealing what's in your URL bar.
If you don't use wildcard certs all of your subdomains can be scraped from the certificate transparency logs. Additionally, any domain+cert using HSTS with preload enabled end up in a big list at Google to speed up the initial connection from browser to site.
The tin hat guess. Did you include Google analytics embedded in the pages? Do you navigate to pages and Google analytics sends that data home? 10 years ago I discovered that Google analytics would send the equivalent amount as organic users; meaning if we sent an email newsletter with links to articles, Google would send almost 1:1 ratio the same number of people from search results. They are tracking everything and using it for more than just reporting.
Do you use a CMS or other tools that auto generate sitemap.xml? Perhaps you unknowingly told Google about those sub-pages.
Google Chrome used to report visited pages back to Google, not sure if this still the case. Also, Google Analytics can see visited pages and Google uses it.
Finding domains is easy, everybody uses CTL to find them.
Google uses data from chrome. If you visited it with chrome, google knows it exists.
I have about 50 subdomains. One was used by a colleague who cant do his shoelaces without claude. That subdomain gets 10 times more spam and hacking attempts.
They log all DNS requests made to their public resolver in a searchable internal database, at least when I worked there a decade or so ago. I wonder if they seed their crawler with it?
Nothing you enter into an LLM not hosted by you, or put onto the web is safe from being collected and exploited by these "AI" companies and their LLM's voracious appetite.
Isn't leaking browser extension used by one of people on the team (doesn't need to be developer, could be qa or anybody with whom the access was shared) more plausible?
You ISP also collects and sells data to companies like Moz, and possibly to Google too.
I've always wondered if Chrome leaks these URLs too.
Hmm. Seems like this could be used for an ARG
I've gotten unguessable hits in the logs because somebody who was authorized to use them had a virus that exfilterated their browser history.
Might have been an evil chrome extension, but ever since Google went IOK2BE ("It's OK to be Evil"), maybe it's just Chrome itself.
It's indexed some unlisted draft blog posts of mine that were never touched by AI or published anywhere. I use a static site generator so there's no earthly way they ever found the pages by scraping, at most I visited the pages once or twice from my browser.
Chat programs catch links you send.
Also that browser setting to check urls are safe sends them out “sometimes“.
I've also seen Google indexing pages with random values in the path that don't get linked to statically (server asks for the URL then redirects to it immediately). I'm pretty sure they index straight out of the Chrome address bar.