logoalt Hacker News

8cvor6j844qw_d6yesterday at 11:13 PM4 repliesview on HN

Does this bypass their own anti-AI crawl measures?

I'll need to test it out, especially with the labyrinth.


Replies

jsheardyesterday at 11:55 PM

They say it doesn't: https://developers.cloudflare.com/browser-rendering/faq/#wil...

Further down they also mention that the requests come from CFs ASN and are branded with identifying headers, so third party filters could easily block them too if they're so inclined. Seems reasonable enough.

xhcuvuvycyesterday at 11:26 PM

Yeah, that'd be huge, like 90% of my search engine results are just cloudflare bot checks if I don't filter it out.

mdasenyesterday at 11:48 PM

If this does bypass their own (and others') anti-AI crawl measures, it'd basically mean that the only people who can't crawl are those without money.

We're creating an internet that is becoming self-reinforcing for those who already have power and harder for anyone else. As crawling becomes difficult and expensive, only those with previously collected datasets get to play. I certainly understand individual sites wanting to limit access, but it seems unlikely that they're limiting access to the big players - and maybe even helping them since others won't be able to compete as well.

show 1 reply
canpanyesterday at 11:28 PM

I feel there is a conflict of interest here..

I'm split between: Yes! At last something to get CF protected sites! And: Uh! Now the internet is successfully centralized.