alt Hacker NewsThis is a crazy change. I wonder if part of the reasoning is that sites without a robots.txt tend to be very low-quality. Search is a very hard problem and in a world of LLM-generated internet, it's become way harder.
Replies
franze • yesterday at 7:50 PM
its not true, just a seo posting stuff and another person thought it was legit
dangus • yesterday at 8:37 PM
Honestly, not crazy. This is should have been how it always was. Why should search engine crawling be opt-out rather than opt-in?
My take: google marketing found a ploy to make "google" look like a better nettizen than the AI companies that hammer away on sites to the level of a DDOS attack.