alt Hacker NewsLLM bots are the easiest to spot. Rate limit them to 16kb/s and 1 request every 5 seconds. Give them a friendly multilingual WCAG status 429 page that says to slow down or status 444 and drop them into nftables ipsets if they continue to be aggressive. Ask your favorite LLM how to set this up. If your LLM gets session paused, ask Gemini or Grok. Tell it if you are behind a CDN. Enable logging of ssl curves, ciphers, etc... Show your LLM some examples from your access logs.
Replies
I get tens of thousands of IPs making 1 or 2 requests against pages on The Consensus (job search, code viewing, etc.). They're not super expensive pages. But I don't want to deal with the bot traffic (mostly because it messes up view counts I use to sell sponsorships). But then if I require an account (even a free one), valid readership is also going to drop with the new friction.
I could build something more complex to discover the patterns, or use Turnstile maybe. I'm not really sure at the moment.
I think you overestimate many organizations ability to implement this type of rate-limiting.
LLM bots make only a part of traffic and really easy to set up (and I think this will eventually throttled by OAI or Anthropic themselves).
But there are bad actors - bots that try every possible urls and traversals and cross scripting and so on… I see it in our access logs. This cannot be solved otherwise than some greylisting/blacklisting on ISP level. E.g. this is so huge that we have started to block all Chinese inbound traffic. I am sorry for that because Deepseek and other good actors are probably blocked as well but IP rotation made it impossible to filter bad ones only… (and I suppose that DS uses EU or US addresses as well so they shouldn’t be blocked in full).