How to Block Some of the Bots

If you are unable to read this, there is an archived copy at https://archive.ph/d3236

You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago.

Block on traffic volume and request frequency if that's causing a problem.

(And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)

I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.

I expect >99% of my web traffic is bots or agents and I was actually considering removing the page counter as it is pretty meaningless and makes my site look far busier than it is. I am reluctant to however do anything about it just in case it accidentally stops a genuine human reading it or downloading my books.

410 and a "Sec-Fetch-Mode:" string in the response body. I guess it thinks I'm a bot? Thanks!

Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)

The guy is certainly kind of visibly ravaged by the more shady denizens of the global internet.

But there are some fun things to read there in any case.

and all valid traffic too, judging by these HN comments (and my own attempts to connect).

what's up with those response headers? "adult" ...

"ai" ...

response length 68

Blocks firefox's built in VPN.

The post content is great. I personally hate the way Cloudfare has been the „default answer“ for the bot problem because Cloudfare has become the most successful MITM attack in history. We need content like this to keep the internet alive.

The added explanations by the author in this comment thread are hilarious. You sir are a good writer.

most (all?) of those will 100% block valid traffic too

There's a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz's, media, etc.), basically anything i might need to wget on a server.

Am I the only one that exclusively gets attacks with spoofed user agents and rotating TLS signatures? I feel like every post I see about not needing a CDN has tips that could be overcome in under an hour of scripting.

[flagged]

[dead]