I documented some crude methods that can stop most of that without a CDN. [1] There will be some false positives so I guess it depends on ones priorities which methods if any to implement or test on a throw-away test site. Not perfect, nothing is. I am watching hundreds of bots sending SYN's and the daemons are oblivious to them. The only method I have not played around with yet is #7 ssl fingerprinting.

There are additional methods I chose not to document such as limiting access to logged in accounts that require double-opting-in to acceptable use policies and terms of use, not that most scrapers would give a toss. That it too much whack-a-mole for me personally. That method requires progressively adding friction to account creation and that comes with some pros and cons.

[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...

> at scale, to the degree that you're taking down my site

Fair. Scrapers should be polite and do their utmost to consume the smallest possible amount of resources.

> without my consent by masking your user-agent

Your consent is not required. It's my user agent. I set it to whatever I want.

> for the purpose of stealing data I didn't authorize you to have

Data can't be "stolen", only copied.

You set up an HTTP server that literally sends people the data when they request it. Don't do that if you don't want people to have the data. Secrecy is the only possible defense.

I think the key word was "intrinsically".

It's like public photography, it's intrinsically legal, except when it's a Flock camera and then it's suddenly an invasion of privacy.

Unfortunately the response to that right now is to nuke everything that isn't within a strict set of constraints. That helps with the bad bulk-scrapers, but hits everyone else as well as collateral damage.

Something like Anubis in front of the server to protect it might be an option. It sucks that we have to resort to that yes, but it seems the least bad option currently (better than the entire internet going through Cloudflare at least).

The idiotic "stealing" argument again? At least use "piracy" if you want to be correct...

The moment you openly publish information on the Internet, you have already given consent. There are other solutions to bandwidth usage.

User-agent discrimination should be illegal. All it does is further the control that Big Tech has, and help authoritarian governments with their control too.

What data is being stolen? Are you referring to copyright violation or something else? If you don't want LLM companies to scrape a site then just restrict access to authorized users. Simple.