An update on residential proxies and the scraper situation

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.

I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.

I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.

Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?

Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)

One article mentioned in the OP was discussed here:

Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)

The issue with scrapping is the intensity and volume of bots.

I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.

Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.

I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.

Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.

The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?

> widespread scraping of web sites in search of training data for large language models and related projects

This is a good thing, thanks to this we have powerful open source LLMs.

> This activity overwhelms sites with traffic.

When LLMs get good enough, we won't need those sites anymore :)

[not satire, this is what I think, without self-censorship]

mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.

The question is more about why the US and others can't properly enforce the bullshit all this amounts to.

Can BitTorrent’s architecture contribute anything useful here?

I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.

There is a large community of people that poison scrapers.

The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.

Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...

[dead]

Again, why do we allow China on the Internet?

Backbone operators should not be allowed to knowingly maintain connections to networks that allow connections from China or Russia.