alt Hacker NewsWebsites should have a lean markdown or .txt page for each human friendly webpage. A lot of the surge in bots is because of LLMs. Its insane that a technical documentation web page can use 200MB + of memory, when the core information I care about is << 1 MB of text. at the path of least resistance for many people is to have claude code hit 20 of such pages.
This is something that would be perfect for cloudflare to host and sell as a service - static web pages via their CDN network.
I do not work in web development, so im sure there are plenty of details im ignorant of, but the TLDR of "how to fight accidental DDOS because of AI tooling " is make it easier for them to get the content they want.
Problem is there's no way to ensure that the bots will only suck in the bot friendly file instead of scraping the whole site. The bot owner would never be able to guarantee that the site doesn't have data not in the bot friendly file. Any time you have to maintain two sets of files is pretty much guaranteed that one will be out of sync at some point. Not all sites are driven by a database that generates files on the fly.