logoalt Hacker News

viraptortoday at 12:43 AM2 repliesview on HN

Have you got any proof of Google scraping from residential proxies users don't know about, rather than from their clearly labelled AS? Otherwise you're mixing entirely different things into one claim.


Replies

misirtoday at 1:23 AM

That's the whole point. Websites that try to block scraping attempts will let google scrape without any hurdle because of google's ads and search network. This gives google some advantage over new players because as a new name brand you are hardly going to convince a website to allow scraping even if your product may actually be more advantageous to the website (for example assume you made a search engine that doesn't suck like google, and aggregates links instead of copying content from your website).

Proxies in comparison can allow new players to have some playing chance. That said I doubt any legitimate & ethical business would use proxies.

idiotsecanttoday at 1:18 AM

I don't think parent post is claiming that Google is using other people's networks to scrape the web only that they have a strong incentive to keep other players from doing that.

show 1 reply