There are also some pathological bad use cases out there that are caused by the combination of trying to scrape all the content, a randomized param, and the scrapers trying to be anonymous. The one I've encountered is pbpBB.

phpBB will assign you a randomized session if if you visit it anonymously (without logging in), and will serve the page to you with that sid param added to all links. Using that link will load a page without the sid param appended to links, because the system assumes you have a session I'd saved in a cookie since it saw you have a sid, even I'd you don't have/didn't send a cookie with it. Following any of those links starts the chain over if you aren't sending a cookie along with a sid, which the AI scrapers never do so they can't easily be blocked by that, and a new unique sid is generated, and all the links have the new sid appended.

Every page with a unique set of params and values needs to be investigated, because it can be an entirely different page, so AI scrapers will scrap forever, always finding more and more URLs to harvest, even though they're all duplicates of what they already have. It's easy to have multiple scrapers stuck going through a very small bulletin board continuously.

Maybe they've gotten smart enough to deal with this situation in the last few months, or maybe they put in a specific workaround. I ended up just throwing the site behind cloudflare because spending time trying to outsmart AI agents in a battle of escalation doesn't seem like an effective use of my time.