alt Hacker NewsDude, I had a 1Gb of nginx logs daily then I blocked them and it dropped to 10mb weakly.
Replies
There are also some pathological bad use cases out there that are caused by the combination of trying to scrape all the content, a randomized param, and the scrapers trying to be anonymous. The one I've encountered is pbpBB.
phpBB will assign you a randomized session if if you visit it anonymously (without logging in), and will serve the page to you with that sid param added to all links. Using that link will load a page without the sid param appended to links, because the system assumes you have a session I'd saved in a cookie since it saw you have a sid, even I'd you don't have/didn't send a cookie with it. Following any of those links starts the chain over if you aren't sending a cookie along with a sid, which the AI scrapers never do so they can't easily be blocked by that, and a new unique sid is generated, and all the links have the new sid appended.
Every page with a unique set of params and values needs to be investigated, because it can be an entirely different page, so AI scrapers will scrap forever, always finding more and more URLs to harvest, even though they're all duplicates of what they already have. It's easy to have multiple scrapers stuck going through a very small bulletin board continuously.
Maybe they've gotten smart enough to deal with this situation in the last few months, or maybe they put in a specific workaround. I ended up just throwing the site behind cloudflare because spending time trying to outsmart AI agents in a battle of escalation doesn't seem like an effective use of my time.
30GB a month isn't too terrible? I guess it depends - for example my provider includes 10TB egress per month, so 30gb would be negligible if it's static content. I'm not sure what others do, but I often click on the links that the agents pull up as sources so the websites do get more discoverable.
What really sucks is e.g. Gemini that pulls images without being able to go to the source (unless it changed recently.)
Edit: I didn't read "logs"
Legit what one of our clients problem was. Normal web traffic wasn’t an issue and the logs were just a few megs. Yeah there was a problem where they were not rotating but once meta started crawling them 24/7 ignoring the robots.txt it kept filling their vps up. An annoying problem to fix once a server is completely full and you can’t even ssh in.