logoalt Hacker News

estimator7292 • yesterday at 7:33 PM • 4 replies • view on HN

What I'm hearing is that if I tweak robots.txt I can exclude my site from google? Excellent news!

Replies

aendruk • yesterday at 7:40 PM

Was that not possible before?

sofixa • yesterday at 7:44 PM

This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it.

➕ show 1 reply
dazc • yesterday at 8:30 PM

no-index is a thing.

➕ show 1 reply